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Aba  tract 

Thie  report,  prepared  as  an  aid  to  the  Quality  Control  (QC)  Project,  summarises 
the  contents  of  sotse  60  documents^  that  vere  thought  to  have  possible  bearing 
on  QC  for  computer  sys tests,  especially  progressing.  Sixteen  of  the  documents 
are  selected  as  worthy  of  further  study,  either  for  valuable  insights  Into  the 
QC  problem,  or  because  they  contain  concrete  suggestions  *r  experimental  data* 

A  synthesis  of  ths  sixteen  is  planned  for  s  future  paper* 
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for  outside  distribution.  Sane  papers  contain  in  their  designation  «h* 
aynbol  (L),  which  indicates  that  their  distribution  is  Halted,  i.e., 
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General  Discussion 


This  document  was  prepared  as  part  ->f  a  project  to  explore  the  possibilities 
of  applying  quality  control  techniques  for  Improving  the  design  and  production 
of  computer-based  information  processing  systems ,  and  more  specifically  for 
improving  computer  programs.  Most  people  who  have  studied  the  subject  agree 
that  measures  of  merit,  except  at  a  primitive  level,  are  lacking  in  the 
computer  software  field;  we  know  when  a  program  fails  tc  work,  but  not  vtoat  Is 
needed  to  make  it  "work  better"--nor,  Indeed,  what  the  expression  "work  better" 
means  in  any  operational  sense.  They  also  agree  that  if  we  are  to  produce 
"better"  software,  we  must  first  find  a  meaning  for  the  word  "better"  in  this 
context;  and,  second,  find  some  way  of  reeo@ii2irig  degrees  of  merit,  so  that 
effort  can  be  guided  in  the  right  direction.  Anyone  undertaking  to  study 
these  problems  naturally  wants  to  know  whether  any  significant  work  has 
already  been  done;  or,  if  not,  at  least  whether  other  people  have  thought 
about  the  problem  and  arrived  at  any  useful  conclusions.  ThiB  document 
consists  of  a  set  of  reviews  of  some  60  papers,  reports,  articles,  and  books, 
all  selected  for  their  possible  bearing  on  the  subject  of  quality  control  in 
information  processing. 

A  true  "Review  of  The  Literature"  was  obviously  impossible,  because  of  the 
wide  dispersion  of  possible  sources:  these  would  include  (among  others) 

Journals  in  Russian,  Japanese,  Chinese...,  together  with  ln-house  papers  and 
graduate  theses  almost  without  number.  Availability  was  a  primary  criterion 
for  selection.  This  led  necessarily  to  a  concentration  on  SDC  and  RAND 
reports.  Such  a  concentration  Is  perhaps  not  as  objectionable  as  one  might 
suppose.  One  correspondent  in  another  organisation  to  whom  Z  wrote  for 
possible  leads  replied,  "I  would  expect  SDC  to  be  the  best  source  of  this  kind 
of  information."  Apparently  SDC  has  the  reputation  of  being  a  pioneer  In  the 
software  field.  Many  people  at  SDC  have  expressed  interest  in  the  quality 
control  problem  and  have  written  reports  presenting  their  ideas,  and  at  least 
some  of  these  authors  have  presumably  been  conversant  with  current  progress 
elsewhere.  But  even  within  8 DC,  it  has  not  been  possible  to  review  every  one 
of  the  thousands  of  documents  produced  during  the  company's  history.  Papers 
were  selected  for  review  because  (a)  somebody  recommended  them;  (b)  the  title 
contained  suggestive  words  (like  Quality  Control,  or  Performance  Measures); 
or  )  the  author  of  the  paper  was  known  to  be  working  on  some  allied  problem. 
The  use  method  of  selection  served  also  for  non- SDC  publications. 

The  total  contribution  of  all  these  papers  is  small.  Rot  one  of  them  is  In¬ 
dispensable,  and  all  of  them  put  together  fail  to  provide  a  firm  basis  for 
further  work.  The  reviews  here  presented  have  mainly  a  negative  value:  they 
may  save  a  future  researcher  the  labor  of  acquiring,  reading,  and  discarding. 
If  he  wants  to  review  some  literature,  he  will  at  any  rate  not  have  to  review 
this  particular  literature.  If  he  decides  to  review  some  of  this  particular 
literature  anyway,  he  may,  perhaps,  be  able,  by  looking  over  lie  reviews,  to 
pick  out  papers  more  likely  to  reward  his  efforts  than  papers  selected  at 
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random,  or  because  they  have  attractive  titloe.  Selection  by  title  can  be 
especially  hazardous;  some  cf  the  least  valuable  capers  reviewed  actually  use 
the  words  ''Quality  Control"  in  their  titles. 

I  have  said  that  all  the  reviewed  documents  put  to  gether  do  not  provide  us 
with  a  firm  basis  for  further  work;  by  this  I  mean  that  they  are  of  no  help. 
Many  of  them  are  nearly  or  quite  unrelated  to  our  task.  A  few  contain  good 
discussions  of  the  need  for  quality  control,  but  these  deal  with  the 
destination,  not  with  ways  for  getting  there.  An  even  smaller  number  give 
charts  of  trails  that  have  been  tried  and  abandoned;  but  in  no  case  is  the 
infomation  complete  enough  to  allow  us  to  assert  confidently  that  a  parti¬ 
cular  trail  can  be  dismissed  as  impossible.  Not  one  of  the  trails  described 
was  followed  long  enough,  and  persistently  enough,  to  establish  it  as  either 
promising  or  unpromising;  in  feet,  one  might  almost  say  that  in  each  Instance 
the  explorer  selected  a  path,  followed  It  for  a  short  distance,  found  that  it 
did  not  lead  to  a  paved  superhighway,  and  then  turned  back. 

But  I  do  not  say  that  the  collection  of  reports  is  valueless.  Some  of  them 
explain,  quite  convincingly,  why  it  is  important  for  us  to  cross  the  mountains. 
They  do  not  help  us  find  a  way  across,  but  they  do  help  us  to  choose  among  the 
four  possible  courses  of  action.  Others  tell  about  approaches  that  the 
authors  thought  might  be  worth  trying;  these  are  speculative,  but  the  specu¬ 
lations  of  informed  people  are  entitled  to  consideration.  Among  the  documents 
we  find  a  few  charts  of  trails  actually  followed  for  a  short  distance.  I  have 
selected  sixteen  papers  for  separate  presentation  and  discussion;  these  are 
reviewed  in  Appendix  A.  In  my  opinion,  these  sixteen  contain  everything  of 
value  to  he  found  in  the  entire  collection.  This  is  not  to  say  that  the 
rejected  papers  contain  no  usefel  ideas,  but  only  that  these  ideas  are 
discussed  adequately  (and  I  think  better)  in  the  selected  group.  There  is 
necessarily  some  repetition  even  within  the  sixteen;  this  is  good,  as  it 
suggests  seme  agreement  among  the  authors. 

Appendix  B  contains  reviews  of  all  the  other  documents  examined.  Most  of 
these  reviews  are  brief,  but  a  few  are  longer,  presenting  as  much  of  the 
original  as  seemed  interesting  or  useful.  The  report  ends  with  Appendix  C,  an 
Index,  alphabetical  by  authors,  of  sll  the  papers  reviewed. 
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Appendix  A 


The  Sixteen  Valuable  Reports 


The  following  is  a  listing  of  the  sixteen  valuable  reports.  In  three  cases, 
several  reports  are  considered  together  as  a  group.  The  order  of  listing  is 
chronological,  beginning  with  the  earliest  date.  For  groups,  the  date 
assigned  is  the  date  of  the  earliest  paper  in  the  group. 


Page  6  Interviews  vith  8 DC  Management,  Corporate  Management  System  Staff, 
FN-6860/0OO/0O, 7 ‘September  1962. 

7  Progranming  Languages  and  Standardization  in  Command  and  Control, 

J.  P.  Haverty  and  R.  L.  Patrick,  RAKD  Document  RM-344t-PR, 

January  1963 . 

8  Management  Aspects  of  Computer  Programming  for  Command  and  Control 
Systems,  V.  LaBolle,  8P-100Q/000/02,  5  February  19^>3>  Management  of 
Computer  Programming  for  Command  and  Control  Systems:  A  Survey, 

K.  Heinze,  n7  Claussen,  and  V. LaBolle,  TM-90 3/000/02,  8  May  1963; 
Quality  Control  in  Comixiter  Program  Development,  V.  LaBolle,  an 
undated  paper  which  appears  to  be  a  draft. 


t 


11  A  Summary  of  Lessons  Learned  te cm  Air  Force  Management  Surveys f 
AS8CP  375-2,  1  June  19^3* 

11  Program  and  System  Testing  (Chapter  16  of  a  projected  book), 

N.  E.  Willmorth,  N- 197^9/370/00,  28  April  1964  (the  first  section 
is  titled  "Program  System  Quality  Control"). 

14  An  Approach  Toward  Quality  Control,  P.  V.  Mclsaac  and 
F.  D.  O^Connor,  N-ZX(L)-&21/000/00,  4  June  1964. 

18  Factors  that  Affect  the  Cost  of  Computer  Programming:  A  ^Quanti¬ 
tative  Analysis,  L.  Farr  and  H.  J.  Zagorsk! ,TM-l447/00l/00, 

31  August  1964. 

18  A  Technique  for  Improving  the  Management  of  a  Computer  Installation, 
R.  L.  Patrick,  RAHD  Memorandum  RM-4232-PR,  September  1964. 

19  Training  and  Job  Performance  Validities  of  Prognumner  Trainee 
Selection  Variables,  04*2172  >  and  Validity  of  Programmer  trainee 
Selection  Variables,  01-2173*  both  by  ballis  it.  KnyT  Aaied 
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Plage  20  Computer  Statue  Quo,  R.  L.  Patrick,  8P-19^7»  15  December  196U. 

21  The  Correlation  of  Computer  Programing  Quality  with  Testing  gffort, 
A.  E.  Tucker,  1M-2219/000/06,  26  January  196$. 
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Interviews  vlth  8 DC  Management .  Corporate  Management  System  Staff, 
Kf-6860/000/00,t  September  1962. 

This  document  presented  management  personnel  vlth  a  series  of  questions, 
and  on  page  1*,  ve  read  Question  3>  part  D,  "Hov  vould  you  try  to  convey 
to  management  the  quality  of  the  vork?"  This  question  Is  clearly 
relevant  to  our  subject,  since  any  attempt  to  convey  the  quality  Bust 
Imply  some  sort  of  estimate  of  quality.  And  on  page  9,  ve  have  some 
answers;  I  quote  nov,  the  bottom  paragraph  on  page  9-  "Question  3-D 
asked  for  vrys  of  conveying  quality  to  management.  In  slight  variance 
vlth  the  intent  of  the  question,  most  interviewees  were  Irst  of  all 
concerned  vlth  ways  of  measuring  quality.  Many  expressed  concern  over 
the  difficulty  of  even  determining  the  quality  of  softv  *e.  Suggestions 
for  improving  quality  control  at  8 DC  also  appeared.  All  responses  to 
this  subquestion  vere  treated  as  belonging  in  a  single  major  Information 
category." 

On  page  11,  there  Is  another  question:  "Have  corporate  goals  been 
clearly  established  against  vhlch  the  contract  can  be  weighed?"  And 
on  page  12,  tvo  more  questions:  "Is  the  proper  skill  mix  available 
vithin  8DCT"  and  then,  tvo  lines  further  down:  "Do  ve  have  available 
the  proper  skill  mix?"  vhlch  seems  to  be  the  same  question.  On  page  13: 
"Can  quality  be  reasonably  measured  and  reported?"  On  page  Ik:  "Are 
reports  available  to  Insure  that  the  most  effective  control  can  be 
exercised  over  the  project?”  Farther  down:  "Are  adequate  control 
measures  being  employed? "  and  still  further:  "Have  proper  criteria 
been  established  to  Judge  the  adequacy  or  the  product?"  In  the  body 
of  the  paper,  percentages  are  given  of  the  number  of  management 
personnel  that  shoved  some  concern  with  the. question.  On  one  table 
vehave  from  88  to  100  percent  of  the  management  personnel  concerned 
with  quality.  In  another  table  ve  have  from  0  to  36  percent.  On 
page  21  ve  read,  about  the  middle  of  the  page,  "A  summary  cessment 
reflecting  frequent  stated  opinion  is,  'Generally,  ve  at  8DC  produce 
too  many  reports,  and  they  are  too  complex.  Ve  need  simple  summary 
reports  vhlch  pinpoint  key  facts  and  serve  as  a  oasis  for  epaclal 
requests  for  detailed  reports.'  Eight  conents  specifically  supported 
the  position  that  8 DC  needs  to  eliminate  unnecessary  reports  oar  parts 
thereof."  Then  another  quote,  farther  down  the  earns  page:  "One 
significant  observation  voiced  by  several  cautions  that  regular  reports 
in  their  present  fora  are  not  the  general  source  of  decision  praises— 
that  information  for  decision  making  generally  stems  from  personal 
contact  or  ad  hoc  information  reporting.  The  regularly  vritten  report 
in  standard  format  becomes  a  useful  historical  document  but  does  not 
serve  to  keep  the  manager  informed  of  the  true  status  of  tbs  project 
at  the  working  level." 

On  page  51,  there  le  a  selection  of  statements  from  interview  data,  and 
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I  quote  a  fev  of  these.  'Project  reports  /et  to  he  bothersome  without 
management  rapport  or  feeder'*."  A  o'  n ,  "Vhen  lower  management  feels 
that  the  information  is  actual  the  quality  of  the  infor¬ 

mation  will  improve."  And  .v.  p-.fcj  >•.  more  Interview  cements:  "If 
you're  on  schedule,  on  budget,  and  your  customer  ia  happy,  this  may  be 
the  best  measure  of  quality."  And  again,  "Management  should  be  more 
concerned  with  quality  and  subordinates  would  find  ways  to  report  on 
quality  If  they  thought  management  were  interested.”  On  the  last  page, 

"I  can  recall  no  other  Instance  of  an  attempt  to  obtain  systematically 
from  within  SDC  at  levels  lower  than  the  managsment  council  suggestions 
for  the  Improvement  of  anything."  Further  along:  "Somebody  should  coma 
up  with  a  measurement  of  quality  and  quantity  other  than  money  spent.” 

It  seems  fairly  clear  that  a  good  many  of  the  people  interviewed  here 
were  concerned  with  the  quality  problem  In  the  sense  in  which  we  are 
talking  about  it,  that  Is,  the  control  of  quality  during  production 
rather  than  the  repair  or  the  correction  of  errors  already  produced. 

Two  or  three  of  the  interviewee  comments  are  to  the  effect  that  if  top 
management  displayed  a  really  lively  concern  about  quality,  the  produc¬ 
tion  personnel  would  find  some  way  to  report  It.  This  has  certainly 
been  the  case  in  other  establishments;  as  long  as  the  general  feeling 
was  that  top  management  was  not  Interested  particularly  in  some  aspect 
of  the  work,  that  aspect  was  likely  to  be  neglected  by  production 
personnel.  I  think  perhaps  this  observation  may  be  the  most  valuable 
pointer  we  can  get  from  this  particular  document. 

Programing  languages  and  Standardisation  in  Command  yd  Control, 

J.  P.  laverty  and  R.  i.  Patrick,  *MNt>  Document  January  1963* 

This  document  belongs  to  the  small  group  that  propose  definite  measures, 
and  moreover,  has  the  additional  characteristic  that  it  Includes  seme 
experimental  data.  On  page  2k,  Table  I,  "Comparison  of  Compile  and 
Execute  Times,”  we  have  some  data  tending  to  show  that  JOVIAL  programs 
take  about  twice  as  long  to  compile  as  FORTRAN  programs,  and  that  the 
JOVIAL  execution  time  at  best  is  equal  to  the  FORTRAN  time,  and  at  worst 
takes  four  times  as  long.  On  page  57;  Table  II,  "Comparison  of  IBM  650 
Programming  Systems,”  we  have  a  comparison  among  four  languages,  with 
compile  or  assembly  times,  time  required  to  load  program  in  computer, 
number  of  chi-square  solutions  per  minute,  using  the  computer  output; 
and  a  similar  problem  (chi-square  solutions  per  minute)  without  read 
or  punch  instructions. 

On  page  31,  we  read,  "The  most  glaring  deficiency  in  the  software  area 
is  in  performance  parameters.  This  deficiency  will  remain  until  we 
develop  a  cost  and  data  collection  endeavor  and  rigorously  define  each 
process  and  subprocess  in  the  programing  area.  In  the  absence  of  these 
definitions,  already  complicated  interrelationships  become  indescribable* 
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We  must  be  able,  at  some  point,  to  analyte  multiple  criteria  and  complex 
performance  trade-offs." 

In  my  opinion,  this  document  definitely  belongs  to  the  small  class  of 
documents  that  must  be  reserved  for  close  reading  by  anyone  interested 
in  applying  quality  cc  trol  to  systems. 

The  next  three  papers  are  treated  as  a  unit  because  of  a  common  authorship. 
These  are  Management  Aspects  of  Computer  Programing  for  Conaand  and  Control 
Systems,  V.  LaBolle,  8P-1000/000/02,  5  February  1963;  Management  of  Computer 
Programing  for  Comnand  and  Control  8ystems:  A  Survey,  K.  He  in  tie,  If.  Claussen, 
and  V.  LaBolle,  TM- 90 3/000/02  ,  3  May  19^3;  finally,  an  undated  paper 
which  appears  to  be  a  draft;  it  does  not  bear  an  author's  name,  tut  it  is  by 
V.  LaBolle  and  has  the  title  Quality  Control  in  Computer  Program  Development. 

The  first  paper,  8P-1000,  deals  mainly  with  costing,  but  since  (in  my 
opinion)  cost  control  and  quality  control  are  very  nearly  opposite  sides 
of  the  same  coin,  it  is  appropriately  included  in  this  review.  LaBolle 
measures  program  output  mainly  by  number  of  instructions,  but  on  page  15 
says  as  follows,  "Although  these  charts  provide  some  Insight  into  pro¬ 
graming  costs  and  their  relationship  to  product  sice,  the  charts  have 
limited  use  for  planning  a  large  programming  effort.  Even  if  these  data 
were  highly  reliable,  and  a  sufficient  number  of  cases  were  available  fcr 
high  confidence,  the  variables  are  missing  that  would  permit  a  manager 
to  make  accurate  cost  estimates  for  programing.  Data  are  needed  to 
establish  the  relation  between  requirements  and  the  sice  of  the  basic 
product,  l.e.,  number  of  Instructions.  Ways  must  be  sought  to  assign 
measures  to  requirements ,  such  as  complexity  of  the  data-processlng 
tasks,  sice  of  the  data  bate,  and  expected  response  time.  These 
measures,  in  turn,  may  be  correlated  with  number  c i  instructions  in  order 
to  find  which  of  these  can  be  confidently  used  as  predictors. 

"A  caveat  is  in  order  with  respect  to  the  use  of  the  variable,  number 
of  Instructions  or  program  site,  as  an  exclusive  measure  of  the  product. 
The  popularity  of  sice  as  a  measure  is  really  based  almost  entirely  cm 
its  availability.  Other  very  important  measures  are  needed  to  indicate 
the  value  of  a  program:  measures  of  (piallty,  e.g.,  freedom  from  error; 
performance,  e.g.,  program  operation  time;  and  convenience  or  'usability1 
are  Important  to  the  customer.  An  obvious  ahortcosdng  of  program  alee 
as  a  measure  of  value  la  that  clever,  experienced  programmers  can 
generate  the  same  logic  with  fewer  Instructions  than  those  needed  by  in¬ 
experienced  pro  gravers.  Even  if  pro  grimmer  capability  were  a  constant, 
the  variation  in  logical  power  of  various  machines  and  their  order  codes 
may  distort  comparisons  between  efforts  In  terms  of  program  site.  Often, 
in  developing  large  program  systems,  computer  storage  is  at  a  premium 
and  programming  effort  la  used  to  reduce  the  number  of  Instructions. 
Therefore,  despite  the  availability  of  program  alee  as  a  basis  for 
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coopering  programs,  dangers  exiau  when  It  Id  used  without  qualification." 
Further  on,  on  page  17,  LaBolle  enumerates,  "Some  other  attributes  ’of 
program  design  that  might  be  defined  more  -'recisely  and  possibly 
measured,”  and  lists  modularity,  versatility,  flexibility,  coupling 
capability  (the  ease  of  adding  new  logic  to  the  program  and/or  ease  of 
integrating  it  into  a  program  system)  and  usability  (the  ease  with  which 
personnel^ other  than  the  designer  can  use  the  program). 

Although  these  comments  relate  primarily  to  program  costing,  yet  obviously, 
a  costing  program  is  meaningless  unless  we  have  a  measure  of  output;  if 
the  output  is  twice  as  great,  then  a  cost  of  twice  as  ouch  would  not  be 
out  of  line.  Ibis  measurement  of  output  is  the  crux  of  the  quality  control 
problem,  since  it  depends  on  the  identification  of  output  criteria. 

The  second  document,  TM-903,  begins  with  an  abstract,  which  states  In 
part,  "Managers  have  difficulty  in  controlling  and  planning  programming 
efforts  without  precise  and  detailed  cost  data,  standard  performance 
measures,  and  definitions  of  tasks  and  products.  Knowledge  of  managing 
and  developing  computer  programing  systems  must  be  extended  and  detailed, 
and  programing  must  be  formalised."  On  page  1  of  the  document,  there  is 
a  list  of  papers  which  the  authors  regard  as  relevant,  and  on  page  2,  a  <• 
remark  that  Is  especially  pertinent:  "Project  members  had  difficulty  In 
gathering  accurate  and  complete  numerical  data."  I  think  this  difficulty 
Is  going  to  hamper  all  attempt*  to  improve  understanding  of  cost  relation¬ 
ships  and  quality  control  until  management  decide*  that  thase  are  prime 
questions  on  which  substantial  research  effort  should  be  expended. 

Generally  speaking,  we  should  not  expect  program  supervisors  and  machine 
operators  to  take  time  out  to  supply  data  for  research  unless  the  effort 
has  some  sanction  at  the  top  management  level. 

On  page  10,  we  read,  "Opportunities  are  plentiful  for  personnel  to  secure 
Jobs  with  Increased  salary  and/or  challenge,  making  it  difficult  for 
programming  managers  to  keep  their  experienced  people."  Further  down 
the  page,  "High  personnel  turnover  impairs  work  continuity."  This 
point,  it  seems  to  me,  has  not  been  sufficiently  recognized  by  other 
authors.  Many  qualified  observers  have  stated  that  the  level  of 
technical  writing  in  the  United  States  is  poor.  If  we  have  a  high  rata 
of  turnover  among  progreumaers,  and  if  they  are  not  very  skillful  at 
documenting  their  work,  the  inevitable  outcome  is  that  much  of  their 
work  is  lost.  I  have  in  mind  in  particular  the  documentation  on  the 
CCIittlfD  Model,  which  was  produced  in  the  ARTA  project  from  January  19^3 
to  November  I96U.  Everything  considered,  I  think  the  documentation  on 
this  project  was  rather  better  than  average;  and  yet,  I  believe  it  is 
not  nearly  good  enough  to  enable  later  workers  to  pick  up  the  project 
where  it  was  laid  down.  If  my  conjecture  is  true,  then  all  the  work  that 
want  into  the  COMMAND  Model  is  in  effect  lost;  most  of  the  personnel  have 
been  terminated  or  transferred  and  future  personnel  attempting  to  perform 
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a  similar  task  will,  in  my  opinion,  prefer  to  do  it  over  rather  than  try 
to  reconstruct  it  from  the  documentation.  In  this  connection,  it  is 
pertinent  to  refer  to  pages  13  and  14  of  the  document  now  under  consider¬ 
ation  ( IX- 90 3) ,  in  particular  Section  C,  which  has  the  headings  "Improved 
Documentation,  Production  Control,  Quality  Control,  and  Information 
Retrieval." 

1M-903  contains  a  number  of  tables  and  other  presentations  of  data 
gathered  in  connection  with  earlier  contracts,  and  so  belongs  to  the 
group  of  papers  that  present  facts  rather  than  those  Uiat  are  chiefly 
speculative.  Moreover,  the  section  beginning  on  page  13  titled  "Opinions 
and  Reccmeendatlone,"  contains  some  fairly  specific  proposals. 

The  consents  on  the  third  paper  will  be  brief,  partly  because  most  of  the 
Ideas  presented  there  are  Included  In  TM-903,  and  partly  because  the  paper 
has  not  been  published  and  so  Is  not  available.  Nevertheless,  a  few 
<|uestlons  and  comments  seem  to  be  In  order. 

On  page  i4,  "The  development  of  improved  quality  control  may  be  regarded 
as  an  iterative  process.  Feedback  from  attempts  to  institute  Improve¬ 
ments  will  lead  to  recognition  of  (l)  what  can  and  cannot  be  controlled, 
(2)  whac  techniques  are  beet  and  (3)  whether  the  effort  is  paying  off.... 
With  respect  to  organisational  structure,  the  clear-cut  assignment  of 
responsibility  and  authority  for  various  aspects  of  quality  control  is 
desirable.  On  the  other  hand,  quality  control  goals  and  methods  must  be 
universally  understood  and  agreed  with.  Improved  quality  control  Is 
almost  impossible  without  mass  participation. ..  .Simple  technique!  used 
consistently  can  improve  quality  control." 

In  this  document,  LaBolle  makes  a  proposal  of  special  Interest,  consider¬ 
ing  the  importance  of  good  documentation  to  the  preservation  of  work  in 
systems  and  programsing.  On  pegs  18,  "Documents  could  also  be  rated  for 
understandablllty,  by  using  a  scheme  such  as  that  developed  by  Fleech 
(The  Art  of  Readable  Writing,  R.  Fleech,  Collier  Books,  1963,  paperback). 
Using  stapling  or  l60  percent  Inspection,  Flesch  shows  how  to  measure 
readability  In  terms  of  interest  and  ease  of  reading.  Interest,  for 
example,  Is  a  function  of  short  sentences  and  the  use  of  many  syllables." 
An  alternate  to  the  Flesch  index  is  that  proposed  by  Robert  Ounnlng  in 
a  booklet,  "Row  to  Take  the  Fog  out  of  Writing,"  published  by  the 
Dartnell  Corporation,  Chicago  1*0,  Illinois,  copyright  1959*  This 
booklet  of  61*  pages  seems  a  good  kind  of  thing  for  the  Corporation 
to.  supply  to  all  personnel  involved  with  documentation,  even  if  the 
proposed  Index  is  not  used. 
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A  Summary  of  Lessons  Learned  frum  All  rn:->  e  Mi'.ns  eement  Surveys ,  ASS  CP  375-2, 
1  June  19^3* 


This  Is  an  official  publicmi.-'n  01'  the  Air  Force.  On  page  57,  ve  have 
a  subject:  Quality  Assurance  Audits  not  Effective.  Among  symptoms  of 
basic  deficiencies  we  find  listed,  "quality  procedures  not  followed, " 
"Production  frequently  failed  to  meet  contract  specifications,"  "lack  of 
cooaunication  between  quality,  reliability,  engineering,  manufacturing 
and  test  personnel,"  "Inadequate  manning  of  quality  assurance  organisa¬ 
tion,"  "incomplete  support  of  top  management,"  "quality  audit  results 
not  enforced,"  "lack  of  follov-up  to  evaluate  effectiveness  of  corrective 
action,"  and  among  desirable  actions  by  contractors,  "increased  staffing 
of  quality  audit  department,”  "QA  trend  charts  and  trend  levels 
established,”  "duality  review  board  established,"  "continuous  surveil¬ 
lance"  and  "provide  for  Air  Force  feedback  of  malfunction  data." 


This  document  appears  to  be  applicable  mainly  to  hardware  procurement, 
but  the  type  of  difficulties  listed  under  the  quality  assurance  heading 
sure  those  that  would  be  associated  with  any  quality  problem.  To  my  way 
of  thinking,  many  of  the  points  made  in  this  document  in  connection  with 
quality  assurance  could  be  carried  over  into  the  quality  control  of 
software,  if  ve  had  some  sort  of  basic  approach  to  the  evaluation  problem. 
Obviously,  the  Air  Force  document  does  not  provide  us  with  this. 

Fromm  and  System  Testing  (Chapter  16  of  a  projected  book),  N.  E.  Vlllmorth, 

1- I§?b9/ yf6/o6 ,  28  April  1964  (the  first  section  is  titled  "Program  3ystem 

Quality  Control"). 

With  respect  to  its  coverage  of  the  quality  control  problem,  this  document 
is  the  longest  and  the  most  comprehensive  of  the  documents  examined. 

From  page  1,  I  quote  the  following:  "Unless  a  comprehensive  plan  for 
insuring  product  quality  is  adopted  that  Insures  continuous  product 
review  throughout  the  process,  adequate  quality  cannot  be  expected  of 
an  ultimate  product  that  is  often  poorly  defined  and  whose  attainable 
performance  is  uncertain  and  involves  considerable  innovation  or  risk. 
Further,  production  of  a  largo  program  system  is  so  expensive  and  lengthy, 
that  the  only  real  alternative  to  a  faulty  product  is  either  to  forego 
the  system  entirely  or  to  live  with  an  inefficient  and  ineffective  system 
until  it  can  be  replaced.  Although  the  greatest  Interest  in  quality 
assurance  lies  in  such  ultimate  products  as  programs  and  operator's 
manuals,  quality  is  not  Just  something  to  deliver  to  a  customer,  but  is 
part  and  parcel  of  the  pride  in  workmanship  associated  with  a  responsible 
and  professional  attitude.  Therefore,  a  quality  control  program  cannot 
be  based  upon  a  single  acceptance  test  or  even  a  series  of  program  tests, 
but  upon  a  continuing  program  of  quality  assurance  from  the  initial 
statement  of  system  requirements  to  the  final  shakedown  of  the  program 
system  in  an  operating  environment.  Quality  must  be  assured,  not  Just 
of  programs,  but  of  the  morass  of  intermediate  plans,  designs,  and 
documentation  leading  up  to  and  supporting  them. 
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"Although  a  good  program  system  development  team  automatically  does  the 
best  Job  it  is  capable  of,  it  is  the  management's  task  to  establish  the 
criteria  of  quality  for  both  products  and  performance  and  to  set  forth 
the  procedures  that  will  make  sure  that  these  goals  are  being  attained. 

Mot  only  must  adherence  to  performance  requirements  and  product  charactsr- 
istlcs  set  forth  by  the  customer  be  observed,  but  to  the  convocations  and 
standards  adopted  by  the  project  to  Insure  greater  efficiency  in  production 
and  maintenance  of  the  product.  Some  standards  are  fixed  by  the  con¬ 
straints  rad  limitations  of  the  production  system  (computer’  and  utility 
system)  and  some  by  the  criteria  of  good  programing  practices  end  the 
state  of  the  art.  Finally,  criteria  and  methods  must  be  set  for  the 
detection  of  the  legions  of  minor  clerical  and  logical  errors  that  almost 
invariably  slip  through  to  crop  up  from  time  to  time  during  system 
operation.  Management  *  s  main  tasks,  then,  are  making  sure  that  precise 
and  accurate  quality  specifications  exist,  that  searching  Inspection, 
and  review  procedures  are  established  and  observed,  and  finally,  that 
the  importance  of  doing  good  vork  is  stressed  in  the  goals  of  the  project 
and  in  the  evaluation  of  performance." 

The  foregoing  quotation  is  the  entire  opening  section  titled  "Program 
System  Quality  Control."  Villmorth  goes  of  with  sections  titled 
"Adherence  to  Specifications, "  "Adherence  to  Programming  Principles, " 
"Adherence  to  Standards,"  "Constraints  and  Limitations,"  and  "Logical 
and  Clerical  Errors."  Hs  then  has  a  section  titled  "Integrating  System 
Quality  Evaluations" :  'Insuring  the  quality  of  such  a  monolithic  product, 
composed  as  it  is  of  largely  conceptual  and  logical  components,  is  not  an 
easy  task.  Traditional  statistical  sampling  techniques  do  not  apply;  each 
product  must  be  reworked  until  it  does  meet  prescribed  standards  or  is 
acceptable  to  the  customer.  Further,  it  is  much  easier  to  apply  quality 
control  techniques  to  individual  components  than  it  Is  to  the  overall 
system,  often  suboptimizing  these  at  the  expense  of  the  system. ...The 
steps  that  must  be  teken  in  setting  up  a  comprehensive  quality  control 
F*ogrem  are  (a)  identify  those  products  whose  quality  must  he  assured.... 
lb)  identify  quality  inspection  procedures  and  tests. .  • . (d)  identify  ths 
points  whsrt  inspections  and  tests  are  to  he  made....(e)  asign  firm 
responsibility  for  conducting  quality  reviews  and  for  testifying  that 
standards  have  or  have  not  been  observed.... (f)  specify  performance  and 
quality  standards  for  quality  reviews  and  reports.. ..As  for  any  other 
product,  conventions  and  standards  should  hs  sat  forth  for  the  format 
and  content  of  review  reports  and  forms »" 

In  e  sot  ions  that  1  mediately  follow,  we  ha  vs  a  subsection  titled 
"Developing  Product  Quality  Criteria":  "Quality  control  plane  must 
specify  procedures  for  reviewing  specifications  as  they  appear  and 
either  phrasing  ths  specification  s tat  meant •  such  that  required  perfor¬ 
mance  end  structural  criteria  are  evident  or  extracting  from  the 
specifications  the  levels  of  quality  that  must  hs  observed.  ••  .Quality 
orlteria  are  In  central  nors  than  a  specification  statement;  both  a 
desired  state  and  allowable  tolerances  should  be  detexmined." 
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In  the  next  section,  under  Quality  Assurance  Procedures":  "It  ife 
important  that  at  least  a  mininun  oet.  or  procedures  be  established  as 
a  guide  and  to  encourage  adequate  r cvx  tvs. ...  Unless  the  test  procedures 
and  requirements  are  quite  clear  a.*i  velx  structured,  much  redundant 
testing  may  be  done  and  testing  is  likely  to  be  as  comprehensive  as 
it  should  be."  The  next  sectio:  titled  Performance  Evaluation,"  says, 

"A  frequently  neglected  aspect  of  the  quality  control  program  is  the 
establishment  of  appropriate  performance  standards  for  quality  review 
personnel. .. .Quality  control  programs  should  establish  criteria  for 
adequate  reviews  and  procedures  for  review  evaluations.  Management 
must  take  note  of  such  procedures,  since  performance  evaluation  is 
largely  a  management  concern,  and  use  them  to  encourage  and  enforce 
adequate  product  reviews." 

This  document  (which  contains  pages)  continues  in  very  much  the  same 
vain,  presenting  the  reasons  why  quality  control  is  necessary  and  the 
objectives  that  a  quality  control  program  ought  to  attain.  There  are 
numerous  flow  diagrams,  and  some  examples  of  what  Willmorth  regards  as 
possible  procedures:  for  example,  on  page  48,  we  find  "A  Sample  Assembly 
Tast  Plain, "  which  details  the  purpose  of  the  test,  identifies  the  programs 
used,  and  gives  the  inputs  and  outputs.  On  page  66,  Willmorth  discusses 
"Organizing  for  Program  Quality  Control,  "  and  says,  among  other  things, 
"If  quality  control  is  to  be  at  all  effective,  it  must  have  the  whole¬ 
hearted  support  of  the  project  management.  It  is  top  management  who 
determines  objectives  sets  policy  and  defines  the  scope  of  project 
activities.  It  provides  the  impetus  in  establishing  product  and 
performance  standards,  and  generates  the  energy  with  which  these  are 
enforced.  In  top  management  lies  the  ultimate  responsibility  for 
quality  and  from  there  stems  the  authority  for  insuring  that  quality 
la  enforced.  If  project  management  neglects  its  duties,  efforts  to 
establish  and  enforce  goals  of  quality  at  a  lower  level  are  built  upon 
a  base  of  sand.... It  must  be  recognized  that  a  quality  assurance  program 
la  not  Just  a  way  of  checking  up  on  subordinates,  but  a  methodological 
approach  to  the  organization's  assuring  itself  that  the  desired  quality 
does  exist." 

It  does  not  seem  practicable  to  quote  at  any  greater  length  from  this 
rather  long  docisnent.  The  general  flavor  is,  I  think,  fairly  represented 
in  the  extracts  given.  I,  for  one,  do  not  quarrel  with  the  author's 
statements;  they  appear  to  present  the  problem  and  define  the  objectives 
as  completely  as  need  be.  Nevertheless,  the  document  still  leaves  open 
the  question,  "How,  in  practice,  are  these  desirable  objectives  to  be 
attained?"  We  need,  in  short,  an  engineering  approach  to  the  problem; 
one  that  carries  out  the  steps  called  for  by  Willmorth.  One  of  these 
steps,  for  example,  is  to  set  up  quality  criteria  and  devise  procedures 
for  insuring  that  these  criteria  are  satisfied.  What  we  need  now  is  to 
perform  this  operation  in  a  specific  case.  Willmorth  points  out  that 
basic  quality  requirements  should  come  from  specifications,  either 


25  March  1965 


Ik 


m-231 3/000/00 


> 


directly  or  by  interpretation.  We  need  to  take  an  actual  cpeclfloatloo 
and  extract  from  it  the  quality  criteria,  and  see  just  how  thla  la  to 
be  done. 

The  next  three  documents  are  tied  together  by  a  coupon  concept  and  a  coupon 
author ahip .  The  firat  of  these  is  An  Approach  Toward  Quality  Control, 

P.  V.  Melsaae  and  F.  D.  O'Connor,  H-DC  ( L ) -621/ 000/06 ,  4  tane  1^64.  fbe 
main  characteristic  of  theae  documents  ia  that  they  propose  a  definite 
kind  of  device,  namely,  "system  catalog"  as  an  adjunct  to  quality  control* 

I  quote  from  the  earliest  of  the  documents.  The  opening  paragraph 
reads:  "The  establishment  and  conscientious  employment  of  sound 
quality  control  techniques  Is  as  necessary  to  the  production  of  a 
computer  base  system  as  to  the  more  conventional  applications  of 
manufacturing.  Quality  control  techniques,  however,  need  not  depend 
upon  strict  statistical  sampling  and  anlysis.  Control  may  be  achieved 
In  a  variety  of  ways,  Including  both  concrete  tools  (e.g.,  COMFOOL,  HRT 
charts)  and  sound  methodology  (e.g. ,  state-of-the-art  techniques).  The 
end  must  not  be  confused  with  the  means.  The  end  is  to  obtain  quality 
through  control.  The  means  must  only  achieve  this  end  within  the 
bounds  of  practicality  and  efficiency. 

"Hew  system,  however,  pose  new  problems  which  necessitate  the  development 
of  new  controls.  Toward  this  and,  the  following  device,  which  we  term  a 
system  catalog,  has  been  conceptualised.  The  system  catalog  will  serve 
for  the  collection  and  description  of  system  data*  Several  of  its 
primary  advantages  are  envisioned  as  follows:  (l)  provide  an  accurate 
and  up-to-date  description  of  all  system  inputs  end  outputs,  (2)  provide 
a  crosscheck  between  operational  design  and  the  real  world,  (3)  provide 
a  means  to  Interact  with  outslda  agencies,  (k)  provide  a  source  of 
Information  for  changes  to  the  program,  ( 5)  provide  for  accurate  inter- 
program  coomunlcatlon.  (6)  provide  guidance  and  control  for  the  efficient 
structuring  of  data,  (7)  provide  a  description  of  the  relationship  between 
operational  deslgi  and  program  design,  (8)  provide  a  convenient  source 
for  the  rapid  updating  of  crucial  changes ...  .It  Is  hopsd  that  a  reallstlo 
attituds  will  ba  assumed  toward  the  catalog  In  order  that  It  may  become 
an  efficient  and  fruitful  tool  rather  than  an  awkward  and  uselesa  relic. " 

There  follow  some  diagrams  in  which  the  system  catalog  Is  shown  as  an 
Intermediate  between  the  ops  equipment  requirements  and  program  coding} 
and  then  there  Is  a  list  of  otaponenta  that  might  be  In  the  oatalog. 
first  we  have  Input*:  "teletype  Inputs,  data  link  inputs,  console 
actions,  keyboard  actions,  and  processing."  Then  wor  outputs,  we  have 
"teletype  outputs,  category  displays,  tabular  displays,  special  displays, 
alarms  bard  oopy  displays." 

There  Is  then  at  the  close  of  the  document  what  the  writer  oalls  "a  sample 
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of  a  single  U25L  system  inpit  meecacv-  and  its  resultant  outputs  as  it 
would  appear  had  it  been  extracted  from  a  complete  catalog."  The  remain¬ 
ing  seven  pages  of  the  document  arc  taken  up  by  this  attachment  and  this 
shows,  in  tabular  form,  the  various  typeo  of  inputs  and  outputs.  The 
table  contains  12  columns  and  each  has  a  heading.  There  are  cross-indexing 
and  various  abbreviations  in  the  table  60  that  it's  necessary  to  consult 
•oas  sort  of  dictionary  or  glossary  to  know  what  the  entries  mean. 

the  second  paper  ie  Initial  Insights  Toward  System  Quality  Control, 

V.  V.  McZsaac,  B-EC-521/lOO/OO,  15  July  1964. 

Again,  quoting  from  the  document,  "Quality  control,  in  its  broadest 
•ante,  refers  to  the  systematic  control  of  those  variables  which  affect 
the  excellence  of  an  end  product.  Quality,  however,  is  an  abstract  word, 
unless  related  to  definable  and  measurable  characteristics  of  the  product 
involved,  in  this  case,  a  programming  system.  But  how  does  one  define 
the  quality  of  a  system?  Obviously,  one  good  measure  would  be  the  degree 
to  which  it  meets  consumer  requirements . . . .Such  a  definition  is,  of 
course,  much  too  general,  for  what  criterion  measures  doee  one  use  to 
measure  adequacy  and  how  and  when  are  these  administered?  Such  are  the 
real  problems  of  system  quality  control.  But  even  at  a  general  level, 
we  can  recogiite  two  distinct  quality  control  requirements:  one  is  the 
need  for  standards  and  specifications  that  establish  the  quality  objec¬ 
tives  to  be  measured  or  evaluated,  and  the  second  is  a  more  dynamic  need  The 

to  provide  the  devices  and  techniques  by  which  such  quality  objectives  p.  V 

are  reached  and  subsequently  maintained. .. .Control  measures  should  first 
be  aimed  at  eliminating  those  assignable  variables  as  opposed  to  chance 
variables  which  might  be  system  availability,  maintainability,  computing 
efficiency  or  reliability.  They  might  also  include  operating  time, 
system  sin,  expansion  capability,  flexibility  to  changss,  or  simply 
a  criterion  that  the  system  be  operational  in  x  months. .. .One  of  the 
greatest  stumbling  blocks  toward  effective  quality  control  in  the  past 
has  been  a  poor  recognition  of  Just  what  was  to  be  controlled. 

"(hie  major  bulwark  standing  squarely  in  the  path  toward  more  effective 
quality  control  is  the  very  conception  of  the  term  itself*  The  develop¬ 
ment  of  a  system  does  not  currently  consist  of  a  single  process  but  is 
more  a  series  of  large  complicated  processes  involving  a  collsction  of 
professionals  whose  needs  for  quality  control  and  thus  their  interpreta¬ 
tion  thereof  are  as  diverse  as  their  backgrounds  and  duties.  System 
quality  control  cannot  be  limited  solely  to  inspection  or  measurement, 
since  a  system,  by  its  very  nature,  must  be  kept  under  continuous  ongoing 
control.  There  is  no  scrap  pile  for  a  system  which  'fails  to  measure  up 
to  inspection  criteria.'  Inspection  is  but  one  necessary  and  admittedly 
neglected  area  among  a  number  of  devices,  techniques,  and  msthodologies 
which  enhance  the  successful  development  of  the  system  and  which  must  be 
relegated  both  legally  and  properly  to  the  domain  of  quality  control. 
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"We  can  break  these  control  needs  into  an  inspection  or  measurement 
function  and  a  control  or  production  function. 

"One  must  readily  accept  the  fact  that  any  'one  set  of  quality  control 
technique  will  necessarily  fail  to  satisfy  an  entire  system' a  needs. 

It  seems  we  must  rather  concentrate  on  control  requirements  according  to 
area  and  function. .. .We  might  attempt  to  develop  techniques  to  arsist 
management  in  controlling  the  production  of  a  system  or  design  p  tool  to 
assist  the  technician  to  better  check  the  logic  of  his  code. 

"A  narrow  definition  of  the  term  quality  control  must  not  serve  to 
restrict  the  means  by  which  system  quality  is  achieved.  All  means 
effective  in  developing  higher  quality  must  be  given  consideration. 

First  consideration,  of  course,  should  be  aimed  at  improving  our  current 
methods  of  operation. .. .It  may  become  necessary  to  revamp  or  even  discard 
old  methods. ..  .This  seems  to  be  the  current  plight  of  the  system  develop* 
ment  process.  Although  we  admit  to  inefficiency  and  lack  of  quality,  we 
nevertheless  are  reluctant  to  adopt  any  device  or  methodology  which  foils 
to  use  the  current  basic  approach  to  the  process.. . .Perhaps  we  should  be  ex¬ 
amining  the  process  itself  rather  than  attempting  to  plug  elusive  holes 
seated  by  the  process  and  tagged  with  the  label  'lack  of  quality  control.8" 

he  third  document  is  Model  of  a  8ystem  Catalog  for  Quality  Control, 

.  V.  Mclsaac  and  F.  D.  O'Connor,  N-UC-(l)-621/200/00,  5  August  1964. 

The  first  paragraph  reads  as  follows,  "This  Note  is  the  third  in  a  series 
of  documents  produced  on  the  subject  of  quality  control  techniques  for 
computer  systems.  It  represents  a  further  development  of  the  initial 
concept  of  designing  an  input- output  catalog  to  be  used  as  the  controlling 
source  for  operational  and  program  system  design ."  The  bulk  of  this  paper 
consists  cf  something  like  24  fold-out  sheets  printed  out  from  a  computer 
and  showing  an  actual  form  that  in  the  authors '  opinion  embodies  the  type 
of  system  catalogs  they  have  in  mind.  This  catalog  apparently  would  con* 
tain  quite  a  sizeable  body  of  descriptive  Information,  and  on  page  4, 
under  subheading  "Using  Catalog  Sections  In  Other  Capacities,"  we  read, 

"In  addition  to  the  conventional  catalog  application  as  a  central  agent 
for  the  collection  and  description  of  system  data,  each  section  of  the 
catalog  might  alao  serve  other  purposes.  For  example,  Section  2,  Data 
Base,  has  been  designed  eo  as  to  retain  the  characteristics  of  the 
symbolic  COMFOOL.  Instead  of  modifying  the  COMFOOL  structure,  catalog 
com.ol  cards  were  added  to  each  table  description.  With  only  trivial 
modifications  to  the  COMFOOL  assembly  program,  this  section  could  easily 
serve  the  dual  function  of  catalog  section  and  symbolic  COMFOOL.  So 
additional  manpower  need  then  be  expended  to  maintain  a  symbolic 
COMFCOL  unique  to  catalog  Section  Two.  Section  Ttiree  might  also  be 
used  in  the  capacity  of  a  set/used  listing,  thereby  reducing  the  need 
for  such  s  function  being  produced  independently  of  the  catalog." 
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The  next  section  is  titled  "Impact  of  s  System  Catalog  on  Existing 
Specifications."  "The  inter.-!  of  the  proposed  system  catalog  is  not  to 
eupplement  the  contents  of  operational  and  program  design  specifications 
tilt  to  constitute  the  primary  and  controlling  source  for  system  outputs 
end  inputs.  It  should  there lore  be  assumed  that  if  a  system  catalog 
evolved  from  the  onset  of  acquisition,  the  current  contents  of  design 
•pecifl cations  would  probably  differ.  The  existence  of  a  system  catalog 
would  obviate  the  need  for  some  of  the  detailed  design  Information  that 
currently  exists  in  operational  and  program  design  specifications.  In 
all  likelihood,  operational  specifications  would  be  subject  to  fever 
changes  (and  inconsistencies)  In  addition  to  being  a  more  meaningful 
document  for  customer  review  and  concurrence.  A  system  catalog  should 
not  be  Just  another  source  of  information,  otherwise  its  primary 
purpose  (to  control)  is  defeated." 

This  is  the  end  of  the  quotations  from  the  three  documents,  all  indexed 
under  the  primary  designator  of  N- LX -621 .  It  seems  to  me  that  in  comment¬ 
ing  upon  these  documents,  it  is  necessary  to  regard  them  in  two  aspects. 
The  first  of  these  aspects,  which  is  best  illustrated  in  document  number 
two  and  in  same  of  the  quotations  from  document  number  one,  deale  with 
the  writer's  concept  of  the  quality  control  problem— its  difficulties 
and  soma  of  the  requirements  of  a  valid  quality  control  effort.  'The 
second  aspect  deals  with  the  proposal  for  a  system  catalog.  In  this, 
the  authors  get  down  to  a  concrete  recommendation,  and  supplement  it 
with  a  rather  extensive  sample  catalog,  presumably  Intended  to  show 
what  such  catalogs  ought,  to  be  like. 

I  can  only  agree  wholeheartedly  with  Mclsaac's  interpretation  of  the 
quality  control  problem,  with  his  philosophy  of  quality  control,  and 
with  his  broad  ldeae  of  how  the  problem  must  be  approached.  I  agree 
that  a  doctrinaire  limitation  to  sow  particular  method  or  control  device 
would  be  a  mistake.  I  agree  that  various  aspects  of  system  production 
and  checkout  will  require  different  methods.  I  agree  that  though  system 
testing  is  Important  and  probably  could  use  such  additional  sffort,  yet 
it  mist  be  distinguished  from  process  control.  I  think  it  perfectly 
possible  that  in  order  to  achieve  this  control,  it  may  be  necessary  for 
us  at  lsast  to  investigate  quite  revolutionary  new  methods  of  system 
production.  In  fact,  I  might  almost  say  that  Nclsaac  has  expressed  my 
opinions  about  the  broad  quality  control  problms  quite  as  will  as  I 
oould  have  expressed  them  myself. 

Vith  respect  to  the  second  aspect  of  these  papers ,  the  "system  catalog" 
device,  I  feel  much  more  hesitant.  I  do  not  see  that  Mclsaac  end  O'Connor 
make  it  clear  hov  this  system  catalog  is  to  be  used  in  practical  quality 
control.  In  fact ,  I  do  not  see  hov  this  listing  of  inputs  and  outputs 
would  help  a  manager  in  any  \my.  The  samples  given  in  documents  1  and  3 
are  not  legible  to  managers  without  special  training;  not  only  would  it 
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be  necessary  to  understand  the  tabular  arrangement,  but  the  person  using 
the  catalog  printout  vould  also  have  to  be  fatal  liar  with  a  considerable 
vocabulary  of  abbreviations  and  conputer  Jargon.  The  catalogs  themselvee 
are  quite  voluminous.  They  do  not  have  the  property  of,  say,  a  quality 
control  chart,  that  the  observer  may  assimilate  a  large  amount  of  informa¬ 
tion  at  a  single  glance;  one  gets  the  impression,  rather,  that  the  catalog 
vould  at  best  be  a  source  of  information  which  vould  then  need  further 
processing  before  it  could  be  presented  as  a  display  to  the  quality 
control  supervisor.  In  short,  it  is  not  at  all  clear  to  me  how  this 
system  catalog  could  be  used,  nor  that  any  substantial  area  of  quality 
control  is  covered  by  the  system  catalog. 

I  acquired  so  high  a  respect  for  the  authors  through  the  general  philosophy 
of  quality  control  as  expressed  in  document  two,  that  I  prefer  to  close 
these  coaaaents  with  reservations.  It  may  veil  be  that  the  system  catalog 
can  be  used  as  the  basis  for  certain  important  control  data  that  vould 
solve  part  of  the  quality  control  problem.  It  may  even  be  that  the 
method  of  its  use  is  implicit  in  the  papers,  and  that  it  is  clear  to 
more  preceptive  minds  than  mine.  The  fact  remains  that  I  do  not  aee 
hov  it  is  to  be  used,  and  I  suggest  that  the  method  of  its  application 
will  need  to  be  made  more  explicit  and  described  in  simpler  terms  before 
it  can  be  generally  understood. 

Factors  that  Affect  the  Cost  of  Computer  Programing;  A.  (friaatltatlve  Analysis, 

L.  ikrr  and  H.  J.  Eagortkl,  TM- 1447/001/00,  31  August  1$6* 

As  the  title  Implies,  this  document  is  concerned  rather  with  the  cost  of 
computer  programing  than  vlth  the  task  of  maintaining  or  assuring  program 
quality.  However,  on  page  11,  the  authors  point  out,  "Cost  estimates  are 
used  for  evaluation.  Squally  important  to  the  direct  uses  of  improved 
cost  predictors  are  the  indirect  uses.  For  example,  predictors  can  be 
sought  that  relate  requirements  and  resources  to  the  methods  used  to 
control  costs." 


Although  TK-lUU?  does  not  address  Itself  particularly  to  the  quality  or 
performance  problem,  yet  in  my  opinion,  it  should  be  part  of  the  library 
of  anyone  working  on  system  quality  and  performance. 


A  Technique  for  Improving  the  Mu agmeat  of  a  Computer  Ins 

r:  r^tH’ffnrc^ 


ter  Installation. 


This,  like  an  earlier  RAID  doeu^nt  by  Patrick  and  feverty,  has  the  merit 
of  recommending  specific  proposals  and  presenting  some  data.  On  page  1, 
in  the  Introduction,  ve  read,  "The  conclusions  of  a  previous  Msmnrsnilitm 
(reference  here  to  the  Ihtrlck-fevarty  paper)  stressed  the  fhct  thet 
there  are  no  generally  accepted  measures  of  performance  for  computer* 
based  systems .  In  the  discussions  thet  followed  publication  of  that 
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Memorandum,  some  concluded  t.iat  any  measure  of  performance  is  sufficient 
to  improve  the  present  state  of  affaire.  An  informal  technique  exists 
and  Is  being  employed  by  some  managers  of  computer  installations,  but  to 
date,  none  of  them  has  formal 'zed  and  documented  the  technique.  It  was 
decided  worthwhile  to  formaJize  the  existing  technique  and  to  make  it 
available  for  possible  use  by  others." 

The  method  proposed  by  Patrick  Is,  however,  only  indirectly  a  method  of 
quality  control;  it  Is  much  more  closely  related  to  cost  control ,  and,  in 
fret,  is  described  by  Patrick  as  a  type  of  cost  accounting.  Patrick 
divides  the  activities  of  a  computing  center  into  2k  categories,  and 
proposes  that  separate  costs  should  be  assessed  in  each  category.  He 
presents  specimen  forms  that  he  thinks  might  be  used,  together  with 
samples  of  how  they  should  be  filled  In. 

Although  the  Patrick  paper  seems  to  deal  with  cost  rather  than  with 
quality  control,  It  may  be  as  well  to  bear  in  mind  that  these  two  aspects 
of  management  are  very  closely  related.  Any  system  that  would  give  us  a 
hatter  understanding  of  costs  would  almost  automatically  give  a  better 
understanding  of  quality.  Although  the  methods  of  assessment  produced 
by  fr trick  do  not  relate  directly  to  quality,  yet  this  document  must,  I 
believe,  be  selected  for  careful  reading  by  anyone  who  proposes  to 
continue  with  quality  control  in  systems. 

TM  papers.  Training  and  Job  Performance  Validities  of  Progreamsr  Trainee 
•election  Variables 7  TH-2112,  and  Validity  of  Programmer  Trainee  Selection 
fariablsa,  TM- 217*37  both  by  Dallis  X.  Perry,  both  dated  9  December  196^ 

These  documents  report  essentially  the  same  data,  TH-2173  being  more  in 
the  nature  of  a  summary  and  more  informal. 

These  documents  are  of  Interest  to  the  present  survey  only  because  they 
include  performance  measure,  which  would  necessarily  be  closely  related 
to,  if  not  identical  with,  measures  of  the  quality  of  a  proyasmer's 
output.  Data  wart  collected  on  U52  trainees,  including  age.  sax,  level 
of  education,  and  scores  on  the  FKA  (primary  mental  ability)  test  and 
the  OCXS  general  Intelligence  test.  Also  available  were  yadss  awarded 
in  training  courses,  first  in  Q-7  programing  and  later  in  8ACB  program¬ 
ming*  On-the-job  performance  was  measured  by  several  criterion  variables, 
including  length  of  employment,  salsa y  progress,  salary  review  rating,  and 
daaeifl cation  progress.  Curiously  enough,  the  largest  correlation  was 
between  eduoation  and  salary  proyeaa  for  trainees  who  took  the  1A0I 
•curse,  and  this  correlation  coefflciant  vaa  ne^tive  and  siyifloant  at 
the  1  percent  level.  Unfortunately,  tb?  relationship  mas  act  confirmed 
with  trainees  who  did  not  take  the  BACt  coux-a*;  tbs  correlation  coeffi¬ 
cient  here  was  again  nsgatlve,  but  equal  only  to  -.01.  If  the  neytive 
correlation  had  oeen  confirmed  for  both  groups,  it  miyt  be  interesting 
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to  try  the  performance  of  person*  vith  only  a  feurth-ff^de  education. 
Several  of  the  positive  correlation  coefficients  attained  slgiiflcanct 
at  the  1  percent  level,  though  this  fhet  oust  be  discounted  when  we 
remember  that  80  such  coefficients  were  computed  for  Thble  8  in  TK-2172. 
The  predictor  that  gave  most  consistent  positive  correlations  vith  the 
four  criterion  variables  was  number  of  dependents,  and  even  here,  the 
largest  correlation  coefficient  was  .23,  suggesting  that  95  percent  of 
the  variability  is  still  unaccounted  for. 

Though  the  correlation  coefficients  obtained  in  this  study  are  not 
exceptionally  small  in  comparison  vith  those  obtained  in  most  validation 
studies,  yet  they  are  ouch  too  small  to  fora  the  basis  for  an  operational 
quality  control  system.  To  put  the  same  statement  differently,  we  do  not 
knov  whether  the  predictor  variables  are  in  fact  related  to  performance 
by  any  strong  relationship,  nor  are  ve  sure  that  the  criterion  variables 
such  as  salary  progress  and  salary  review  rating  are  valid  measures  of 
oo-the-Job  performance.  The  conclusions  of  these  two  papers,  that  the 
teste  proposed  have  some  value  in  aiding  in  programmer  selection,  are 
not  disputed;  but  if  a  quality  control  procedure  le  to  support  day-to- 
day  actions,  a  performance  criterion  le  needed  vith  a  much  smaller 
standard  deviation  than  is  Indicated  for  the  criteria  used  here.  Perry 
obtains  better  correlations,  of  course,  by  using  a  weighted  combination 
of  predictors;  but  this  improvement  would  itself  have  to  be  validated  by 
a  repetition  of  the  experiment.  In  any  case,  It  seems  unlikely  that  the 
perfoamanoe  measures  uoed  by  Perry  are  sufficient  for  routine  quality 
control  purposes. 

Computer  Status  frio,  R.  t.  Patrick,  8P-1947,  15  December  1964. 

Unlike  the  other  document  by  Patrick,  this  one  does  not  contain  any  data* 
It  dose  contain,  beginning  on  page  21,  a  suggested  sequence  of  actions 
for  producing  a  program.  Patrick  divides  the  total  effort  into  eeven 
acts,  each  bf  which  is  further  divided  into  scenes  (Act  VII  consists  of 
a  single  scene.  Act  V  contains  12).  The  tone  of  much  of  the  writing  le 
•omewhat  tongue  in  cheek,  but  it  may  well  be  that  a  discussion  of  this 
kind  could  be  valuable. 

On  pegs  34,  re  trick  begins  an  epilogue,  in  which  he  says  in  part,  "The* 
author  le  not  aware  of  any  concentrated  effort  to  solve  the  debugging 
problem  by  understanding  the  debugging  process  so  that  proper  tools  may 
be  produced  to  alleviate  it.  In  the  past,  we  have  spent  vast  amounts  of 
money  without  e  clear-out  Objective  or  approved  plan.”  Patrick  con  eludes 
his  paper  vith  the  following  sentences:  Ne  may  design  systems  that  are 
so  ilmmnfllng  that  we  osn  never  make  that  work.  Such  Is  the  life  of  the 
computer  professional.  What  an  exciting  time  to  live." 

Although  X  do  not  believe  that  this  paper  cental  as  any  Important 
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■ugf»stlon«  for  operational  quality  control  or  improvement ,  yet  It 
is  readable  and  perhaps  should  be  included  in  the  quality  control 
analyst's  small  shelf  of  books. 

The  Correlation  of  Computer  Programming  Quality  vith  Testing  Iffort, 

A.  V.  Tucker,  Bt-22i9/000/00,  £0  January  1965* 

Among  all  the  papers  reviewed,  there  are  very  few  that  propose  an 
operational  scheme  for  evaluating  any  part  of  program  production,  but 
this  neper  is  one  of  those  few.  Tucker  has  collected  data  on  the  number 
of  discrepancy  report  forms  as  a  function  of  time;  tbs  time,  that  is, 
expended  in  searching  for  and  finding  programming  errors*  There  is  a 
modest  body  of  theory,  applicable  to  problems  of  proofreading  or  search, 
that  might  be  invoked  In  this  kind  of  effort.  TUcker  presents  some 
graphs  shoving  hov  the  number  of  errors  discovered  changes  with  pro¬ 
gressiva  time,  and  in  the  light  of  proofreading  theory.  It  seems  quite 
possible  that  some  mode  of  estimation  could  be  devised  that  would  give 
an  estimate  of  the  number  of  errors  undiscovered  after  e  particular 
time,  using  ea  input  the  number  >f  errors  already  found.  Tucker  says 
in  pert,  on  page  32,  "A  qualitative  feeling  or  indication  of  the  quality 
of  an  individual  program  can  be  obtained  from  the  elope  of  the  error 
acanulatlon  versus  testing  effort  data  plot.  This  feeling  or  indication 
oac  be  reinforced  by  the  use  of  an  estimated  total  error  population  for 
the  program. . . .The  relative  quality  of  different  computer  programs  can 
be  established  by  a  data  normalising  procedure  in  which  the  total  error 
population  for  each  program  is  established.''  TUcker  goes  on  to  recoasmnd 
that  a  more  extended  and  systematic  effort  be  made  to  collect  data  on 
numbers  of  errors  and  testing  iffort,  and  to  analyse  these  data.  "Until 
sufficient  data  is  available  to  justify  other  standards,  the  quality  of 
BFO  ccmputar  products  shculd  be  defined  in  terms  of  e  nominal  model  based 
upon  the  normalised  date  presented  in  this  report." 

It  is  so  refreshing  to  find  a  document  with  a  specific  operational 
procedi  re  to  propose,  that  perhaps  my  enthusiasm  rues  away  with  me; 
nevertheless,  the  number  cf  errors  In  a  computer  program  is  certainly 
ome  measure  of  its  $iallty,  and  there  does  exist  some  statistical  theory 
that  mould  allc.  its  estimation,  following  the  lines  suggested  by  TUcker. 
X  think  this  document  mist  b*  placed  very  high  in  the  stack  of  dooumsnts 
to  be  considered  seriously  in  deciding  hov  to  continue  vith  the  problem 
of  quality  control. 
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Thie  Appendix  contains  reviews  of  all  document*  considered  but  not  included 
In  the  selected  16. 

Computer  Prograc  System  Development  Milestones,  no  author,  8SD  Exhibit  6l-VfA, 
no  date. 

This  docuaent  "defines  the  contend  of  the  computer  progrma  development 
milestones  products  to  be  delivered  to  the  Air  Force. "  It  does  not 
present  specific  methods  of  quality  evaluation,  but  it  does  Indloate 
that  the  Air  Force  expects  soae  sort  of  quality  standards  to  be 
observed.  For  exaaple,  on  page  21,  it  speaks  of  acceptance  criteria, 
"Outputs  checked  and  any  particular  results  to  be  noted  are  checked. 
Possible  data  to  be  preserved  for  other  tests  should  be  Identified." 

Ibis  on  Milestone  5*  There  is  another  requlranant,  under  Milestone  11, 
that  oalls  for  quality  recording  and  reporting,  though  again  vith  no 
suggested  Measure  of  per foraaace . 

The  docuaent  is  of  no  value  for  suggesting  specific  quality  control 
Measures;  Its  only  interest  lies  in  the  vitness  it  beers  to  the  Air 
Fbros's  Interest  in  quality. 

SAGE  Fro  area  TtI  Mentation  History  lo.  2— Bow  to  Set  up  a  Field  Site  (1997). 

8.  g.  fla^fgl^/H'fstsuSy  I&. -  - 

This  docuaent  Is  Included  only  because  the  title  auggsst#  it 
have  Material  relevant  to  quality  control;  in  faet,  It  oontalns  ao 
such  Material.  It  Is  chiefly  an  outline,  apparently  intended  for 
training  instructors,  shoving  hov  they  might  prooasd  at  a  field  site 
to  break  In  nev  personnel. 

The  subject  of  testing  Is  listed,  vith  such  headings  as  "Test 
Fhilosophy,"  Standard  Test  Concept,"  "Additional  Pasting,"  and 
"As sash ly  Test  Procedures."  Operational  testing  Is  described  only 
In  a  very  gaaeral  vay.  In  «y  opinion,  this  docuaent  has  nothing 
of  interest  for  our  case. 

"Bov  Do  You  Measure  Useful  Computer  TlaeT"  Autoamtlc  Data  Processing  Digest, 
▼oluas  6,  Dumber  5,  March  i960,  pages  lfc-17* 

The  sbetrsct  begins  vith  the  sentences,  "Measurassnt  of  good  and  bad 
computer  tlae  Is  not  easy.  Boas  activities  aay  be  considered  good 
(i.e. ,  productive)  and  others- -bad  (or  nonproductive),  vhile  sobs 
aay  be  open  to  doubt  or  retried  ae  neither.  The  activities  are 
grouped  under  these  headings:  preventive  maintenance,  actual 
production  runs,  developasnt  of  pro  grans,  tins  lost  due  to  coamuter 
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fruits ,  repair  time  of  computer  fauVts,  time  lost  due  to  operator 
or  programmer  errors ,  idle  time,  aiace] laneous  occurrences.'’  Further 
alone,  the  writer  points  out  that  there  Is  no  unique  scale  for  the 
measurement  of  goodness  or  badness  in  computer  utilization  and  that 
widely  different  figures  for  the  same  set  of  circumstances  might  he 
obtained  merely  by  applying  different  yardsticks.  Ee  concludes  with 
the  recommendation,  "it  is  suggested  that  manufacturers  get  together, 
possibly  under  the  auspices  of  the  British  Computer  Society,  and  'eettle 
a  formula  that  could  be  used  for  all  installations  and  produce  strictly 
comparable  figures. 

Xt  yearns  clear  that  here  again  we  have  a  pointing  out  that  the  problem 
estate,  with  not  much  specific  in  the  way  of  a  proposed  solution. 

ft  *r31tv  Control,  T.  J.  Snyder,  If- 18476  ,  22  June  1962. 

Mr*  S&yder  begins  his  paper  with  the  following  sentence,  "Perhaps  the 
first  problem  to  be  addressed  in  a  working  paper  of  this  kind  is  that 
of  a  cautious  definition  of  the  term  quality  control.  At  first  blush, 
of  course,  this  tern  does  not  appear  to  be  much  more  vagie  or  difficult 
than  acme  others  we  have  lived  with  and  used,  tut  it  is.  Quality  control 
is  a  dreadful  term  made  worse  by  increasingly  general  use  and  misuse." 
After  scan  preliminary  discussion,  Snyder  resolves  the  problem  Into 
recognizing  two  general  classes  of  errors  which  he  calls  system  errors 
and  nonsystem  errors,  and  the  objective  of  quality  control  appears  to 
be  the  control  of  the  frequency  with  which  these  errors  occur.  Both 
system  end  noneystem  errors  can  occur  as  either  random  or  what  Snyder 
calls  methodic;  I  quote  now,  "Honeys tarn  errors  probably  comprise  the 
bulk  of  any  error  package.  These  run  the  gamut,  from  simple,  foolish 
mistakes  to  great,  irrevocable  blunders.  They  are  discouraging  to 
find  and  worse  to  be  responsible  for.  Of  all  the  types  of  errors  we 
should  most  want  to  control,  we  are  least  likely  to  control  these. 

But  perhaps  we  can  find  the  weens  to  minimise  tbm. 

"tendem  nonsyatem  errors  are  very  bad  things.  Thsy  result  from  keypunch 
failures,  dull  pencils,  tired  eyes,  etc.  These  random  errors  oan  crop 
up  almost  anywhere  in  the  production  cycle,  and  in  any  medium  employed. 
They  are  bad  things."  Further  along  la  the  paper,  we  read,  "Off  line 
error  checking  bears  the  brunt  of  most  so-called  quality  control  opera¬ 
tions.  It  is  here  that  the  most  sophisticated  planning  and  creative 
thinking  Is  needsd,  end  has  always  been  lacking.  Normally,  in  a  highly 
oomplax  system,  this  operation  exists  as  sn  sx  post  facto  produet 
msasuro.  In  fret,  off  11ns  checking  should  be  largely  incorporated 
into  the  pre production  phase;  that  is,  throu(£iout  the  input  specifica¬ 
tion  tad  preparation  stages.  Further,  it  may  be  more  feasible  to  plan 
its  application  in  small  portion-— not  only  to  detect  errors  as  soon 
as  possible,  but  in  order  to  distribute  the  operation  in  time  and 
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•pace."  Finally,  near  the  end,  Snyder  saye,  "Ve  rauat  determine  whether 
we  want  an  error-checking  or  an  error- correction  provision,  or  both, 
incorporated  into  the  program  system.  This  will  be  a  difficult  teak. 

By  adopting,  for  example,  a  set  of  prograaaed  measures  that  will 
accomplish  sophisticated  error  correction  in  conjunction  with  error 
detection,  the  quality  control  feature  may  become  unwieldy.  The  feature 
can  become  so  complex  Itself,  in  fact,  that  it  too  may  begin  generating 
sufficient  errors  to  cone  under  the  Jurisdiction  of  yet  another  quality 
control  feature." 

It  does  not  appear  that  in  this  paper,  Snyder  has  done  more  than  point 
out  the  need  for  quality  control,  and  the  dichotomy  between  measures 
used  after  the  product  has  been  produced  (which  I  prefer  to  call  accep¬ 
tance  inspection)  and  measures  applied  to  the  production  process,  which 
I  regard  as  true  quality  control.  The  paper  contains  interesting 
discussions  of  the  kinds  cf  errors  thst  can  adversely  affect  quality, 
but  does  not  really  suggest  anything  specific  in  the  way  of  procedures. 

On  Measuring  the  Validity  and  Effectiveness  of  the  Response  of  a  Military 

Command  Control  Syatea/cT  K.  Gordon,  Jr.,  W-l5&51/odQ/66,  July  1^62. 

This  document  Is  not  directly  addressed  to  the  problem  of  quality 
control,  but  since  it  deals  with  a  measure  of  effectiveness,  it  is 
worth  considering  in  say  survey  of  supposedly  quantitative  approaches 
to  a  system  problem. 

Z  quote  from  the  first  paragraph,  "Of  the  many  kinds  of  responses  open 
to  a  military  command  control  system,  possibly  ths  most  obvious  is  the 
attack*  Such  a  response  has  a  number  of  different  properties,  two  of 
which  will  be  considered  In  this  paper,  vis.,  validity  and  effectiveness 
•  * .  .The  validity  of  a  response  may  be  defiped  as  tbs  degree  to  which  &t 
response  confoms  to  the  Intent  of  the  commander.  The  effectiveness  of 
a  response  may  be  defined  as  ths  extent  to  which  the  Intent  has  been 
realised."  The  rest  of  the  paper  is  a  somewhat  mathematical  treatment 
based  on  some  rather  abstract  concepts,  and  It  is  not  at  all  dear  that 
the  perimeters  needed  for  the  mathematical  expression  are  operationally 
ascertainable.  In  any  cast,  the  document  contains  no  suggestion  at  an 
Operational  level  that  might  provide  us  with  a  real  measure  of  either 
validity  or  effectiveness.  Gordon  even  says  on  pap  6,  "The  attempt 
to  define  validity  or  effectiveness  in  toms  simple,  universal  way  is 
fraught  with  difficulties."  The  purpose  of  the  paper  seems  mors  to 
mph&slse  these  difficulties,  rathsr  than  propose  any  method  for  over* 
coming  them.  As  such,  it  seems  to  have  little  value  for  our  present 
study  and  so  Z  think  can  be  dlsreprded. 
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The  following  two  papers  are  by  different  authors,  but  on  the  sane  subject, 
so  they  are  discussed  together,  "hase  are  Information  Processing  8ystem 
Paslaa:  General  Principles,  Part  I,  Jack  Jaffe,  TM- 7^3/001/00,  9  August  1962, 
anA  Principles  o£  Informtion  Processing  System  Design:  Part  II, 

George  Masters,  TM- 74 3/002/00,  10  February  1953. 

In  the  first  pap*'*,  Jaffe  says,  on  pege  41,  ’’Evaluation  of  the  Design* 
fhia  is  the  evaluation  of  the  design  product  itself,  as  a  design, 
before  tiie  system  is  constructed  or  before  the  design  process  is 
reiterated  whichever  approach  has  been  selected  by  management* . . . 

Srrors  of  omission  and  internal  consistency  are  searched  out  and  when 
found,  fed  back  to  the  design  production  effort  for  correction.  This 
is  a  hard  and  lengthy  process  and  one  that  a  great  deal  of  time  must 
be  allowed  for.... A  very  important  element  of  the  evaluation  process 
is  the  establishment  of  design  quality  criteria.  This  means  the 
discovery  of  the  important  features  of  the  system  and  eventually 
the  setting  of  levels  of  acceptability.  Particularly  with  computer 
based  systems,  there  is  a  need  for  defining  the  relevant  functional 
areas.  A  programmer  may  need  to  know  access  speed  for  core  memory, 
but  a  designer  needs  to  know  how  long  an  operator  will  have  to  wait 
far  a  new  display  that  he  has  requested." 

In  the  second  document,  beginning  on  page  34,  under  the  heading  "System 
Analysis,"  Masters  says,  in  part,  "In  discussing  system  analysis,  we 
will  not  cover  analysis  involving  mathematics  or  other  sophisticated 
analytic  or  rigorous  techniques  although  these  techniques  are  certainly 
important  aids  to  the  designer  when  they  axe  applicable.  The  concentra¬ 
tion  here  will  be  on  what  will  be  called  rational  analysis."  Masters 
goes  on  to  discuss  the  subject  in  very  much  the  same  way  that  it  is 
discussed  in  the  book  by  Goode  and  Machol,  using  very  much  the  seme 
terminology.  No  specific  procedure  is  suggested. 

Although  the  discussions  in  these  two  papers  are  interesting  and  would 
bt  useful  to  someone  approaching  the  problem  of  system  design  for  the 
first  time,  I  see  no  helpfulness  for  the  task  of  creating  a  methodology 
for  quality  evaluation  or  control. 

the  next  two  documents  are  closely  related  to  each  other,  and  they  are  of 
special  Interest  because  both  of  them  contain  the  expression  quality  control 
in  the  title.  In  point  of  time,  the  earlier  of  these  documents  is  WCft  lffib-- 
hann—  ndatloc.  Real  Time  Quality  Control.  W.  L.  Thomson  and  H.  P.  flows  t, 
Ii-53by 57&/00,  24  August  1$62. 

The  opening  paragrsph  reads  as  follows,  "Ons  of  tbs  principal  criteria 
for  successful  operation  of  the  SAGE  system  is  that  each  subsystem  be 
maintained  at  design  level.  That  is,  performance  of  each  subsystem 
must  be  checked  and  verified  to  Insure  the  reliability  of  that  subsystem 
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as  an  integral  part  of  the  SAGE  system.  Present  techniques  for  accom¬ 
plishing  this  are  inefficient  and  quite  often  ineffective.  This  document 
outlines  modifications  in  SAGE  system  design  to  provide  a  real-time 
quality  control  capability." 

A  little  down  the  page  ve  have,  under  the  heading,  "Recommended 
Solution,”  the  fclloving:  "The  recommended  solution  to  the  quality 
control  (QC)  problem  is  a  computer  program  vehicle  which  is  time-shared 
with  the  real-time  simulation  program.  This  vehicle  consists  of  several 
new  DCA  programs  which,  together  with  modifications  to  existing  DCA 
programs,  will  perform  the  following  functions: 

a.  Real-time  monitoring  of  LRR  and  GFR  test  messages. 

b.  Real-time  analysis  of  SIF  codes  reported  by  MK  space  x  input 
equipment. 

c.  Real-time  analysis  and  correction  of  registration  and  collimation 
errors . 

d.  Periodic  calibration  of  height  finder  RHI  consoles. 

e.  Real-time  monitoring  of  the  height  comnuni cations  loop. 

f.  Real-time  analysis  of  negative  height  replies. 

g.  Recording  of  information  preparation  of  subsystem  performance 
summaries. 

The  implementation  of  some  of  these  features  requires  equipment  modifica¬ 
tions;  additionally,  the  programming  of  the  time-shared  solution  can  best 
be  implemented  so  as  to  be  concurrent  with  other  SFC  activity.  For  these 
reasons,  an  interim  package  which  does  not  contain  monitoring  of  GFR  test 
messages  or  any  of  the  height  features  will  be  provided.  This  interim 
package  will  provide  an  early  operational  quality  control  -apnbility, 
temporarily  using  existing  spare  drum  allocations  vt'icn  will  be  freed  by 
the  release  of  the  final  time- shared  package." 

If  we  look  through  the  remainder  cf  this  document,  we  find  that  it  con¬ 
sists  essentially  of  proposals  for  0  number  of  standard  test  problems  or 
messages  intended  to  be  presented  tc  the  system,  with  monitoring  of  the 
resulting  output.  ThU  is  conceptually  Identical  with  industrial  testing, 
where  the  product  is  checked  from  time  to  time  to  verify  its  performance; 
it  is  distinct  from  process  quality  control,  which  is  intended  to  monitor 
the  process  of  production-  In  the  subject  document,  a  number  of  teete 
are  described,  but  these  art  all  quite  specific  to  the  particular  system j 
they  have  no  enteral  application,  and  could  not  be  transferred  to  compu¬ 
terised  systems  of  all  kinds.  Necessary  as  this  type  of  lnspeetlOQ  and 
oontrol  Is  to  the  eueoeeetui  operation  of  a  large  system,  it  is  not  the 
total  taek  as  that  visualltsd  in  the  present  approach  to  quality  oontrol* 
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Closely  allied  to  this  is  Interim  Production  list  of  SFCR  1576  (the  reference 
is  to  the  previous  document)’,  R.  V,  Green,  N-1?2d4/0^0/02,  15  October  1962. 

H- 17264  appears  to  be  an  attempt  by  the  author  to  translate  the  require¬ 
ments  of  SFCR  1576  into  computer  Jargon.  Thus,  on  page  29,  ve  read  under 
the  heading,  "Alternate  Quality  Control/Real-Tlme  Simulation 


Taken  by  ASO,  AST 

C0  Activate  Button  is  Used 

C02  if  QPTI  is  0,  FSMI  must  not  be  'OK*  (3). 

Rpl  Complement  ARTI 
R02  set  QMOD  to  1" 

and  so  oa»'  This  is  then  a  set  of  instructions  to  the  prograaner 
Intended  to  enable  him  to  carry  out  the  ideas  of  SFCR  1576,  and  as 
such,  is  of  no  general  interest  to  us. 

Oieratiocal  Measures  of  Information  Characteristics,  R.  M.  Lon gel re  and 
§•  J.  Bricks on,  8F-930,  1  September  1962. 

the  title  of  this  document  is  very  encouraging,  but  I  do  not  believe  the 
contents  quite  cone  up  to  the  expectations  generated.  The  authors 
reoonond  a  device  which  they  call  a  8TAMIC  chart,  which  turns  out  to  bo 
■Imply  a  double  dichotomy  in  which  Information  statements  are  classified 
as  true  or  false,  relevant  or  Irrelevant.  This  proposal  is  followed  by 
a  symbolic  mathematical  treatment,  which,  in  my  opinion,  is  not  very 
operational,  in  spite  of  the  title.  Z  am,  In  fact,  not  able  to  see  how 
the  proposals  of  the  authors  could  actually  be  followed  out  in  any 
practical  cate.  Thus,  I  believe  the  document  may  be  set  aside. 

Simulation  for  Bvaluatlng  Product  Quality,  J.  R.  Crawford,  BP- 992, 

U  October  (written  for  presentation  at  the  17th  Midwest  Quality  Control 
Conference  in  Denver,  October  26  and  27,  1962). 


The  title  of  this  document  is  appealing,  but  upon  examination,  it  turns 
out  to  be  a  proposal  for  a  queuing  modal  to  simulate  transactions  in  a 
warehouse.  I  do  not  believe  it  has  any  relevance  to  our  problem. 


jha  Hu 
VOWNMi 


Huelye  Criteria  In  Cosmand  and  Control,  Gordon  M.  Becker,  SB-196, 
r  1$6£,  General  Electric  (TUfro),  Santa  Barbara,  California. 


This  paper  was  presented  to  the  human  factors  working  group  at  tha  Tenth 
Military  Operations  Research  Symposium  In  October  19o2.  Becker  starts 
out  by  saying,  "One  of  the  oldest  probltms  we  fact  in  military  research, 
and  one  of  the  most  difficult  to  rssolve  Involves  ths  selection  of  a 
criterion  to  evaluate  a  military  system,  subsystem,  or  component.  Ths 
problem  is  especially  important,  and  difficult,  in  commend  ooctrol 
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system 0  research.  It  Is  important  because  these  systems  vitally  affect 
the  nation.  It  is  difficult  because  these  systems  must  be  acceptable  to 
many  diverse  groups  in  the  nation  prior  to  being  procure!,  and  because 
they  oust  satisfy  the  requirements  of  a  multitude  of  users  after  they 
are  procured.”  A  little  later,  we  read,  "Given  this  structure  within 
which  we  must  operate,  what  can  ve  do  to  increase  the  efficiency  of  cur 
efforts  in  the  design  of  command  control  systemn?  Two  recommendations 
to  the  Director  of  Defense  Research  and  Engineering,  which  appeared  in 
a  report  of  the  Institute  for  Defense  Analysis  are  pertinent  to  this 
problem.  These  recommendations  were  (l)  mechanisms  whereby  technical 
and  functional  compatibility  efforts  are  coordinated  within  the  Depart¬ 
ment  of  Defense  should  be  strengthened,  (2)  the  military  users  of  a 
command  control  system  should  actively  participate  in  the  design  develop¬ 
ment  and  the  evolution  as  well  as  the  use  of  the  system. 

*Ve  can  help  to  implement  both  of  these  recommendations.  Ve  can  help  if 
we  restrict  the  assumptions  we  make  in  our  studies  to  those  that  are  the 
direct  responsibility  of  the  agency  that  funds  our  study,  l.e. ,  we  should 
not  act  as  though  we  know  the  input,  requirements  or  utilities  of  agsndes 
other  then  the  particular  agency  and  its  subordinate  agencies,  that  are 
directly  responsible  for  our  study."  Again  on  the  next  pegs,  "Ve  can  also 
help  by  making  more  use  of  simulation  during  our  design  work.  This  siaai-  . 
Lation  should  be  so  structured  that  the  contracting  agency  can  participate 
in  the  test  runs.  Moreover,  the  design  group  can  help  the.  contracting 
agency  make  use  of  other  government  and  military  groups,  including  the 
alternate  users  of  the  system  both  *bove  and  below  the  level  of  the  system 
assigned  to  the  agency  to  participate  in  the  simulation."  On  the  last 
page,  "In  summary,  I  have  indicated  that  the  selection  of  e  command 
control  system  depends  upon  the  criteria  of  many  groups.  Since  all  of 
the  groups  in  the  evaluation  chain  met  be  satisfied  If  a  system  la  to 
be  adopted  by  the  operating  forces,  none  of  the  systems  accepted  will  be 
judged  best  by  any  group  if  they  apply  different  criteria*  Moreover,  the 
designer  oannot  expect  to  obtain  an  analytic  expression  of  the  real 
utilities  used  to  evaluate  military  systems.  Despite  the  lack  of  an 
explicit  analytic  utility  expression,  we  can,  by  simulating  our  systems 
in  the  design  phase  and  by  including  other  DOD  a  gneiss  as  subjects  and 
experimenters  in  the  simulation  of  the  system,  desigi  systems  which 
satisfy  the  criteria  of  the  various  groups  in  the  chain  and  increase 
their  cooperativeness.  Thus,  ve  oan  satisfy  that  elusive  nrmmmnit 
control  criteria  through  simulation  studies  even  if  we  oannot  measure 
it."  This  is  the  final  paragraph  of  the  paper. 

It  semis  to  me  plain  that  although  Mr.  Becker  has  spoksn  at  langth 
about  the  need  for  reconciling  the  criteria  of  various  users,  he  has 
at  no  time  got  specific  about  any  of  the  criteria,  nor  about  methods 
far  assuring  that  any  criterion  Is  satisfied*  I  think  the  p^er  has 
no  value  trm  our  point  of  view. 
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iUgt  In  the  Declgi  and  Development  of  an  Information  Processing  System, 
tarrelne  G#  G«y,  Sr-i023.  9  lfoven^r  19&2. 

This  paper  is  included  because  the  title  suggests  that  it  sight  contain 
something  about  evaluation;  in  fact,  It  does  cot. 

the  next  two  papers  are  treated  as  a  unit  because  of  their  common  authorship. 
First  la  Isplaasntlng  Computer  Systems  and  Evaluation  of  Operational  Systems, 
§>-916/002/(30.  10  January  1^63-  The  second  is  Implementing  and  Evaluating 
j^»ihi!jatlop^fro«>ssing  Systems,  8P-1294,  6  November  19&j>7both  by 

these  papers  are  not  identical,  but  they  include  essentially  the  ease 
Material,  and  the  following  quotations  and  ccmnsnts  are  fro®  the  latter 
one,  BP- 1294. 

On  page  2,  ve  read,  "Nov,  the  Introduction  of  data  procaasing  devices, 
suoh  as  computer,  laplles  that,  for  one  reason  or  another,  the  existing 
system  Is  unsatisfactory. . . .The  rtasons  for  adopting  automation  are 
expressed  in  such  gmsral  terms  as  'increased  efficiency.'  It  is  true 
that  one  should  expect  greater  efficiency  of  operation,  but  his  sxpecta- 
tlons  should  he  glared  to  specific  instances  if  he  la  to  properly  assess 
the  potential  benefits  to  ba  derived  from  automation.  These  specific 
Instances  furnish  the  criteria  by  which  a  system  aay  be  evaluated. 

"There  is  nc  single  set  of  criteria  e^lnst  which  all  systems  oan  be 
mas  mire  d  md  which,  at  the  same  tims,  would  be  sufficient  for  the  evalu¬ 
ation  of  u  given  system.  Accordingly,  although  this  paper  discusses 
certain  general  standards,  it  will  emphasis*  the  prooess  of  developing 
particular  criteria  for  individual  systems. "  Developing  this  idea, 
Idsmsrwi  begins,  an  page  4,  a  list  of  questions,  "which  the  system 
dsslgw  mist  ask  himself  about  his  observations  of  user  activity  and 
questions  which  he  must  also  direct  to  the  user.”  Among  these  questions, 
ws  have  one  retarding  user  objectives,  soother  on  operational  functions, 
another  on  operational  tasks,  still  another  on  information  requirements, 
and  so  on,  ending  with  the  question  ”Vho  uses  tbs  information,  and  for 
what  purposes?" 

Cm  page  19,  vs  read,  "The  operational  requirements  furnish  criteria  by 
which  the  user  evaluates  ths  finished  product.  They  also  fUmlsh 
criteria  by  which  the  designer  evaluates  ths  deslgi  specifications, 
which  in  turn  furnish  ths  basis  for  ths  designer's  evaluation  of  ths 
production  effort.  Ttasrs  are  other  more  graeral  ways  of  evaluating 
systems. . .one  of  these  is  costs. 

"There  is  00s  final  criterion  which  demands  close  study  by  the  user: 

...is  ths  equipment  being  used  to  oapaeityt" 
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Thl»  It  about  as  specific  as  Adamson  gets  in  his  discussion  of 
performance  criteria,  and  clearly  not  specific  enough  to  support  a 
•quality  control  activity. 

Managing  the  Development  of  Complex  Systems,  M.  G.  Holmen,  I-1976O, 

TTC^£7l9&3T  - - - 

In  the  Introduction,  Holmen  says,  "There  are  at  least  eight  major  steps 
to  he  performed  in  managing  the  development  of  a  complex  ccsnaad  and 
control  system.  This  article  is  concerned  vith  those  steps,  the  sequence 
in  which  they  are  taken,  and  the  way  in  which  they  are  arranged." 

Further  along  the  eight  major  steps  are  described.  Step  6  includes 
system  testing,  but  this  paper,  again,  does  not  present  any  method  for 
quality  control  of  program  production  or  for  the  evaluation  of  a  produced 
program. 

Detailed  Evaluation  Flan  0100  by  Field  Design  Branch,  TM-OI-003/OOO/OO, 

36  April  1963  (no  author  given). 

This  papsr  was  Included  because  its  title  suggests  that  it  deals  with 
the  evaluation  problem.  Actually,  its  scope  is  quite  narrow.  It  refers 
to  an  experiment  on  system  U25L#  in  which  the  experimenters  sought  to 
determine  the  effect  upon  system  response  time  of  inputs,  additional 
consoles ,  and  recording  programs.  The  conclusions  from  the  expsrlmsnt 
are  not  very  precise:  "It  would  be  extremely  tenuous  to  attempt  to 
•xtrmpolate  these  exact  results  to  changts  in  recording  parameters, 
Increases  in  the  number  of  consoles  and  simultaneous  actions,  or  changs 
of  inputs.... Further  lnvesti^tloa  is  currently  being  planned  along 
these  lines."  Sines  the  experimenters  did  not  seem  to  feel  that  they 
had  established  anything  very  firmly  and  since  at  best  ths  experiment 
has  limited  scope,  there  seems  to  be  no  likelihood  of  further  from 
this  paper. 

The  next  three  pepers  are  treated  as  a  unit  becauss  of  their  common  authorship* 
They  are  Command  and  Control  Management  Decision  Making,  SF-U7^»  20  May  19631 
Development  of  an  Operational  Msnsgsmsnt  System.  6M1t5,  l6  April  1963> 
anA  Social  Value  and  Technologioai  Deal  gal,  CT»i?06,  6  January  196^#  all  by 

it*  frfGsr~™^ 

A  quotation  from  ST-1171*,  on  page  id:  "A  major  difficulty  of  states 
monitoring  is  the  selection  of  appropriate  data,  fciey  managers  cannot 
be  expected  to  absorb  data  describing  everything  about  the  co^eny  nay 
more  *****  an  admiral  or  a  general  can  be  expected  to  spend  altfxt  and  day 
before  the  displays  of  a  ccxmmad  and  control  system.  Ths  data  for  status 
monitoring  should  be  carefully  screened  and  selected  so  that  they  are  of 
interest  to  ths  croup  which  will  see  thmi....The  danger  of  status  moni¬ 
toring  is  ths  well  known  frustration  osussd  by  messes  and  messes  of 
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reports  being  sent  to  manageit  -who  <’o  net  have  time  to  read  them.'* 

Very  nearly  the  same  paraernnh  occurs  in  PP-’175»  page  8«  In  SP-1506, 
page  k,  under  the  heading  "Values  and  Criteria,"  we  read,  "The  values 
toward  which  a  technological  desiga  is  directed  are  those  characteristics 
of  the  ensuing  product  that  are  considered  desirable.  A  criterion  is 
the  degree  of  the  valued  characteristic  vnich  must  be  obtained  before 
the  product  is  acceptable."  (This  is  not,  I  believe,  the  definition  of 
criterion  that  is  usually  accepted.) 

If- 19709/311/00,  G.  F.  Veinvurm,  22  May  1963. 

This  short  document  is  a  preliminary  outline  for  a  chapter  with  the  title 
"Operational  Design."  It  has  a  section  under  the  heading  "What  is  a  good 
operational  designT”  and  this,  I  think,  comes  close  to  our  subject. 
Veinvurm  lists  as  good  qualities  that  an  operational  design  should  have 
accuracy,  consistency,  completeness  and  logical  organization.  Probably 
none  of  us  would  quarrel  with  these  objectives,  but  Veinvurm  does  not 
attack  the  problem  of  hov  these  qualities  are  to  be  measured,  and  that, 

I  believe,  is  at  the  core  of  the  quality  control  problem. 

JOVIAL  Caspiler/OASIS  Maintenance  Procedures,  V.  M.  Mineart,  TM-WD-70, 

57  May  1963. 

The  opening  sentences  of  this  document  are,  "The  purpose  of  this  document 
is  to  describe  the  procedures  used  by  SDC  in  maintaining  the  Phase  I 
JOVIAL  compiler  and  OASIS  systems  at  the  NAVCOSSACT  computer  facility. 
Further,  the  document  provides  guidance  to  the  users  of  these  systems 
in  the  reporting  of  discrepancies  and  malfunctions  of  the  systems  and 
provides  for  the  dissemination  of  information  concerning  the  disposition 
of  errors."  This  paper  is  specific  in  that  it  presents  definite  report 
forms  that  have  actually  been  used  to  report  discrepancies,  which  could, 
of  course, 'be  regarded  as  quality  failures.  However,  it  does  not  at  all 
address  the  problem  of  overall  system  performance  testing,  nor  of  the 
establishment  of  performance  criteria. 

"'Prediction  of  Creativity  in  a  Sample  of  Research  Scientists," 

Cecil  J.  Mullins,  in  IEEE  Transactions  of  Engineering  Management,  June  1963> 

page  52. 

I  will  quote  the  summary  which  appears  at  the  beginning:  "In  an  attempt 
to  identify  test  predictors  of  scientific  creativity,  two  criteria  of 
creativity  vere  ueed:  supervisor's  ratings  and  number  of  publications. 

An  interest  questionnaire,  a  vocabulary  test  and  nine  tests  of  the 
Guilford  Creativity  Battery  were  administered  to  131  research  physical 
scientists.  Of  42  test  scores  derived  from  the  battery,  4  were  signifi¬ 
cantly  related  to  the  rating  criterion  and  7  to  the  publications 
criterion.  The  two  criteria  vere  not  significantly  related  to  each 
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other  and  none  of  the  predictor  scores  correlated  significantly  with 
both  criteria.  A  composite  predictor  gave  promise  of  increasing 
effective  prediction  of  the  ratings  criterion,  tut  not  of  the 
publications  criterion. " 

Although  this  article  is  not  alned  at  quality  control,  it  ms  examined 
because  of  the  possibility  that  a  breakthrough  in  measuring  scientific 
creativity  would  give  some  clue  for  methods  to  measure  program  produc¬ 
tivity  or  program  excellence.  As  appears  from  the  sumsary,  no  such 
hope  can  be  entertained;  ve  are  still  unable  to  predict  creativity  in 
research  scientists,  and  in  fact,  one  might  even  argie  whether  either 
supervisor  ratings  or  numbers  of  publications  is  a  valid  measure  of 
creativity.  The  paper  is  accordingly  laid  aside  as  of  no  Interest  to 
our  project. 

Is  Relevance  an  Adequate  Criterion  in  Retrieval  System  Ivaluatlont 

tsuaren  8.  Doyle,  BP-1266, 1  July  1&3* 

This  paper  ms  Included  in  the  survey  because  the  title  suggests  that 
criteria  and  evaluation  might  be  discussed,  as  indeed  they  are;  but  not 
In  a  quantitative  way  suitable  for  quality  control  uses. 

An  Updated  flan  for  a  Corporate  Mhma— mnt  System.  A.  J.  Rhine,  24  July  1963. 

this  document  Is  conoemed  mainly  with  describing  the  various  activities 
that  have  to  bo  undertaken  in  developing  a  specific  maaagmmmt  system. 
Borne  attention  is  given  to  timing  and  uee  of  the  PSRT  system,  but 
nothing  here  apparently  relates  to  quality  control. 

Ifeaagmmaat  Standards  for  Data  Processing.  Uck  I.  Brandon.  Tan  Bos  timed. 

c#t ft  Set - ~ - 

The  problem  of  quality  control  in  progressing  is  still  far  from  a 
solution,  and  thus  one  does  not  look  for  much  help  in  books,  which  tend 
to  be  n  few  years  behind  the  current  Journal  articles.  Still,  Brandon 
has  two  chapters  labeled  "Methods  Standards:  Prn greasing,"  and  clearly 
theme  suet  be  eamtaed,  if  only  because  of  their  title. 

The  first  of  the  subject  chapters  (numbered  XT  in  the  hook,  pp*  69*109) 
dealt  with  the  standardimtion  of  procedures:  flow  charting  conventions, 
character  writing  conventions,  and  the  like.  The  eeoond  chapter  has 
headings  such  as  "Testing  and  Program  Validation”  (page  110),  "Vesting 
Standards"  (page  115),  and  "Program  Chengs  Administration”  (pegs  143); 
but  none  of  this  material  teams  specific  enough  for  uee  in  quality 
control.  There  ars  also  sections  on  "Performance  Standards,"  which 
propose  such  comparisons  as  "Production  Tlme/ConUlng  Time"  (page  213) 
or  "Average  teet  eet-up  tlme/Average  test  time"  (pegs  220).  Some  of  the 
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suggestions  are  concrete  encash  taoie  page  236)  but  I  get  the 

impression  that  they  were  derived  .’vom  a  contemplation  of  what  ought  to 
be,  rather  than  fron  experi  e:.^c  v  a  working  crew. 

In  any  case,  the  methods  describe.!  ra  nearly  not  been  validated  by 
actual  trial  and  iterative  modification.  One  would  expect  that  paper - 
of  more  recent  date  would  be  more  valuable  for  our  purposes.  I  think 
we  have  to  accept  the  fact  that  we  cannot  get  much  help  from  books. 

Command  Control  Software  System  Development  During  the  Conceptual  Program 
Definition  and  Acquisition  Phases,  A<mTTT^X-74/ooo/o66  Draft,  lV  August  1963. 

This  document  weighs  five  pounds  three  ounces,  including  cover,  and  so 
by  dint  of  sheer  size  might  be  supposed  to  have  something  for  everybody. 
When  it  is  published  in  Its  final  form  the  pages  doubtless  will  be 
consecutively  numbered,  but  l rs  the  draft  most  pages  are  not  numbered 
at  all.  Instead,  they  bear  designators  such  as  Dl- 50-29  or  AS-U9-I,  and 
explanations  of  these  designators  appear  at  the  beginning  of  the  sections 
into  which  the  document  is  divided.  At  the  back  of  the  book  is  a  set  of 
fold«out  charts,  and  these  chow  the  designators  and  their  relationship 
to  the  general  scheme  of  classification. 

Designator  C-25  means,  Preliminary  System  Analysis  and  includes  seven 
numbered  subheads,  including  No.  2,  Prepare  Evaluation  Framework ;  No.  6, 
Evaluate  Cost-Performance  of  Alternatives;  No.’ 7,  Select  Most  Promising 
'Configuration  and  Adjust  System  Performance  Requirement. 

We  have  also  C-25A,  System  Performance  and  Design  Documents. 

Under  C-25. 2  we  read,  "Th,ls  marks  the  first  step  in  a  continuing  concern 
of  the  system  design  engineer  with  the  problem  of  design  verification 
and  validation. 

"Based  upon  a  review  of  interface  consideration,  command  organization, 
mission  and  operational  environment  information  and  the  structure  of  a 
threat,  the  system  design  establishes  a  set  of  general  performance 
criteria  which  will  bs  used  as  a  measure  of  system  concept  ef f activity. " 

C-25 -7  begins  with  the  following  paragraph:  "With  concurrence  with  the 
user,  the  Systems  Division  will  select  the  most  promising  configuration 
for  a  preliminary  system  design  sub-phase.  The  configuration  is  selected 
as  a  result  of  the  previous  cost /performance  evaluation  activity." 

C-25. 6,  Evaluate  Cost  Performance  of  Alternatives,  says,  "Cost  performance 
studies  involve  the  determination  of  the  cost  of  a  certain  specifiable 
complex  of  machines,  devices  and  facilities  which  will  provide  the 
capability  of  implementing  a  specific  system  idea  or  concept  (mission 
concept  or  concept  of  operation).  This  cost  is  compared  with  other 
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costs  developed  from  studies  of  other  configurations  supporting  a 
different  concept  of  operations  (system  concept)  or  other  configurations 
supporting  the  same  system  concept. 

"Included  in  the  items  for  cost  performance  analysis  is  the  traditional 
principle  of  system  configuration;  vhich  in  conm&nd  control  is  the 
geographic  dispersal  of  men,  equipment  and  facilities. 

"We  must  include  in  an  analysis  the  software  or  computer  programing 
element  with  an  implied  examination  of  trade-off  between  human  and 
automatic  information  processing  operations  and  decision  making. 

"For  instance,  the  greater  the  automatic  processing  requirement,  the 
greater  is  the  nrfed  for  equipment  and  programing  capabilities  and  the 
less  control  exercised  directly  by  the  human  decision  maker.  From  a 
cost  point  of  view,  by  reducing  the  number  of  decision  makers  and 
increasing  the  size  of  your  electronic  magnetic  information  processing 
complex,  one  could  be  playing  operations  dollars  off  against  development 
dollars . 

"From  an  operations  point  of  view,  the  question  is  one  of  control  and 
reliability.  Once  the  automatic  information  processing  complex  mal¬ 
functions  or  ceases  to  work,  the  entire  system  comes  to  a  halt.  This, 

In  the  case  of  the  death  or  incapacity  of  a  CIWC  or  senior  officer,  is 
not  true;  the  proper  subordinate  takes  over  the  role  of  decision  making 
and  battle  managmaent.  The  designer  has  to  examine  and  measure  the 
coat/worth  value  of  greater  apeed  and  accuracy  and  capability  for  the 
automatic  processing  complex  against  its  relative  vulnerability  and 
I^ck  of  strength-in-depth. 

"*he  most  promising  configuration  of  men,  machines,  facilities, 
automatic  information  processing  complex  are  selected  and  prepared 
for  the  next  design  step." 

The  shove  quotation  includes  the  whole  of  the  entry  under  C-25.6. 

These  quotations  appear  to  be  the  ones  most  closely  related  to  the 
subject  of  quality  control.  About  all  one  oan  say  of  th«  is  that  they 
concede,  at  leaat  by  implication,  that  quality  control  is  an  Important 
objective.  Mothing  is  said  anywhere  about  methods  or  eonapts  of  approach 
to  the  quality  control  probloi.  In  fact,  about  all  the  document  really 
aays  is  that  we  have  an  obllpatloe  to  deliver  a  reliable  produet  of  good 
quality.  lev  we  distinguish  good  quality  fraa  bad,  or  how  we  are  to  sat 
about  making  a  produet  reliable  remains  to  be  determined;  I  find  nothing 
la  the  proemet  doomtut  that  sheds  light  on  these  questions. 
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jjwgth  In  the  Management  of  Computer  Programs,  V.  LaBolle,  TM-1626/OOl/oo, 
t  December  1963. 

On  page  17,  there  la  the  beginning  of  a  section  under  the  heading 
"Quality  Control  Techniques.-'  "In  this  task,  we  would  undertake  to 
clarify,  define,  and  help  determine  measurements  for  the  quality  of 
computer  programs  and  computer  program  documentation.  A  part  of  this 
area  of  work  that  overlaps  the  cost  work  described  above  would  seek  to 
build  a  classification  system  for  programs  In  terms  of  what  they  do. 

Such  a  taxonomy  is  vital  to  the  cost  analysis  and  la  implied  by  many  of 
the  cost  factors  already  identified. 

"A  deeper  Investigation  of  quality  would  consider:  (l)  what  programs 
are  supposed  to  do  and  how  they  are  Intended  to  be  used  as  reflected  in 
requirements  and  design  specifications,  (2)  what  programs  actually  do  as 
determined  by  test,  exercise  and  operational  use,  (3)  ways  in  which 
desired  queLllty,  including  performance  characteristics,  can  be  expressed 
unambiguously  and  preferably  quantitatively,  and  how  the  products,  both 
documents  and  programs,  can  be  Inspected  during  each  programming 
activity  to  Insure  that  the  quality  standards  can  be  met. 

"Clearly,  the  study  of  quality  can  not  only  contribute  to  insight  into 
a  cost/value  relationship  in  program  development  but  also  will  point  to 
changes  in  the  methods  for  performing  the  programming  Job." 

This  document  is  essentially  a  research  proposal  and  the  section  under 
quality  control  say a  in  essence  that  because  quality  control  is  an 
important  problem  we  should  be  devoting  some  effort  to  its  solution. 

It  recognizes  that  operational  methods  can  be  developed,  if  at  all, 
only  by  long  research,  and  thus  we  do  not  find  in  this  document  anything 
more  than  a  recognition  of  the  gravity  of  the  quality  problem;  there  is 
nothing  suggested  in  the  way  of  specific  methods. 

Organization  Decision  Making,  Julian  Feldman  and  Herschel  E.  Ranter,  SP-1357, 
JO  December  1963 . 

The  authors  consider  the  decision-making  process  as  one  of  selecting  a 
particular  path  among  a  number  of  alternate  paths,  so  as  eventually  to 
reach  a  desired  goal.  Although  the  methods  proposed  are  relatively 
close  to  the  standard  approaches  of  matrix  theory,  game  theory,  and 
operations  research  generally,  yet  the  presentation  here  appears  to  be 
too  abstract  for  isnediate  value  in  the  quality  control  problem.  No 
attention  is  paid,  for  example,  to  the  question  of  selecting  among 
alternate  goals;  it  is  assumed,  in  the  Feldman-Kanter  treatment,  that 
a  goal  has  been  selected. 

Z  think  ve'aay  set  this  paper  aside. 
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A  Comparative  evaluation  of  JOVIAL  and  FORTRAN  IV,  C.  J.  Shaw,  N-21169, 
t  January  1964. 

This  paper  was  included  for  examination  because  its  title  suggested  that 
there  might  be  acme  sort  of  verifiable  criterion  used  in  the  comparative 
evaluation.  It  turns  out  that  this  is  not  quite  the  case.  In  the  paper, 
ve  have  (in  parallel  colvmns)  discussions  of  various  features  of  JOVIAL 
and  FORTRAN,  and  at  the  conclusion  of  the  paper,  statements  such  as,  for 
example,  on  page  23,  "FORTRAN  IV  is  better  for  compiling  large  programs 
out  of  many  small  subroutines. 


•FORTRAN  IV  compilers  provide  for  the  manual  inclusion.  Into  the  programs 
they  process,  of  previously  compiled  subroutines,"  and  so  on.  In  short, 
the  two  systems  arc  described  in  qualitative  lanc^iage;  there  seems  to  be 
nothing  here  that  would  lend  itself  to  quantitative  assessment. 


A  User's  Experience  with  Three  Simulation 


&MPAC),  Karen  Young.  TM-1755/6oo/50,  17  February! 


GPSS,  SUBSCRIPT,  and 


This  paper  was  reviewed  because  its  title  implies  a  comparison  and 
comparison  in  turn  implies  evaluation.  The  author  considered  method 
of  modeling,  programing  flexibility,  outputs  provided,  programming 
time  required,  and  programing  ability  required;  but  all  these  quantities 
were  assessed  subjectively,  and  I  do  not  believe  they  give  us  inch  hint 
for  quality  control  methods. 
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atlonal  Procedures,  L.  A.  Friedman, 


On  the  cover,  this  is  identified  as  a  chapter  to  be  contributed  to  a 
book,  The  Development  of  Computer-Based  Information  Systems,  sponsored 
by  SDC.  Rear  the  end  of  his  chapter,  on  page  166,  of  a  total  of  114 
pages,  Friedman  has  a  section  titled  "Testing  the  Operational  Procedures.” 
Re  makes  an  interesting  point,  on  page  107,  "One  of  the  major  and  most 
important  reasons  for  procedure  testing  is  that  procedures  which  are 
developed  from  design  documents  are  only  nominal,  that  is,  what  should 
be.  Experience  has  always  shown  that  what  actually  is  done  in  an  oper¬ 
ational  environment  does  not  always  resemble  what  should  be  done.  Thus, 
delivering  the  procedures  to  a  potential  user  without  testing  them  can 
lead  to  troubled  operations.  Hence,  a  test  must  be  designed  to  check 
the  validity  of  the  procedures,  i.e.,  do  the  operational  procedures 
prescribed  actually  operate  and  control  the  computer  programs  for  which 
they  are  written?"  There  is  a  little  sort  along  this  line,  but  here, 
again,  the  emphasis  is  obviously  on  testing  a  completed  program  or 
perhaps  monitoring  it  over  a  period  of  continuous  operation;  nothing 
here  related  to  the  control  of  the  program  production  process  itself. 
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Automated  Computer  Efficiency:  the  ACE  Method  for  Efficient  Computer 
fhrogrMBaiag»  M.  I.  Bolaky  and  3.  L.  Felngold,  ~9F-1292/000701^  5  March  1964. 

This  document's  title  suggests  that  it  might  be  relevant  tc  our  problem, 
but  it  is  in  fact  not  ao.  On  page  1,  in  the  Abstract,  ve  read,  "This 
paper  documents  ACE,  a  method  for  the  syotemization  of  the  innumerable 
details  Involved  in  digital  coaputer  programming  and  checkout,  specifying 
these  details  in  step-by-step  form  on  a  series  of  checkout  charts*  This 
system! fl&tion  insures  that  programmer  actually  perform  ell  of  these 
details,  and  in  correct  order.  The  charts  indicate  the  specific  actions 
to  be  taken  to  prevent  errors  and  to  track  dovn  the  causes  of  errors 
that  do  occur.” 

Ve  have  here,  then,  essentially  a  procedural  or  checkout  list,  whose 
purpose  is  to  insure  that  no  essential  step  is  omitted.  It  has,  I  think, 
no  relevance  for  our  task. 

System  Progransing  Management,  N.  E.  tfillmorth,  TW-1 578/000/00,  13  March  1964. 

This  document  is  in  process  of  revision;  the  following  quotation  on 
quality  control  is  from  a  preliminary  version  (page  173): 

"Control  of  document  and  program  quality  is  one  of  the  toughest  problems 
facing  the  programming  manager.  Making  sure  that  documents  are  accurate 
and  programs  are  debugged  takes  so  much  of  the  manager's  efforts  that 
other  aspects  of  document  and  program  quality  are  often  Ignored. 

"Quality  control  of  documentation  and  programs  presents  scorn  unique 
problems.  High  levels  of  reliability  are  demanded,  but  exacting  methods 
of  determining  reliability  and  validity  are  not  well  established.  Since 
the  products  produced  are  one -of-a- kind  items,  sampling  methods  are  not 
applicable.  A  document  or  program  must  pass  all  tests  in  order  to  be 
acceptable.  Fortunately,  quality  criteria  for  most  programs  are  concerned 
with  whether  or  not  the  programs  perform  the  required  function  and  occa¬ 
sionally  with  processing  speed,  but  seldom  with  elegance  or  optimum 
efficiency.  Once  in  a  while  flexibility  and  modularity  are  mentioned, 
tat  criteria  of  flexibility  and  generality  are  seldom  established  or 
enforced. 

"On  the  other  hand,  program  and  program  testing  seems  an  interminable  Job. 
Oetting  a  complex  system  completely  debugged  seems  an  impossible  task* 

Many  programming  managers  would  like  to  establish  some  mans  of  stopping 
testing  activities  short  of  complete  perfection  without  being  called  to 
account  for  every  bug  discovered  in  the  future.  The  determination  of 
seme  method  of  establishing  of  reliability  and  quality  better  than 
debugging  for  obscure  cases  is  desirable. 


25  March  1965 


38 


TM- 2 31 3/000/00 


"In  addition  to  product  excellence  quality  control  la  used  for  the 
evaluation  and  improvement  of  the  processing  system.  Keeping  statistics 
on  the  ’scrap  rate'  of  designers,  programaers  and  computers  is  s  com  thing 
that  is  infrequently  done.  Computer  reruns  are  sometimes  accounted  for 
when  the  rerun  Is  due  to  machine,  system  or  operator  error,  hut  seldom 
for  program  errors.  The  amount  of  'had  code*  that  is  produced  and 
scrapped  in  the  course  of  producing  a  program  is  almost  never  accounted 
for,  although  rates  ranging  from  50  to  500  per  cent  have  been  alleged  for 
particular  programs.  Very  little  can  be  done  to  improve  the  quality  of 
performance  in  program  and  document  production  unless  such  scrap  rates 
and  other  performance  measures  are  collected  and  evaluated  for  normal 
performance  measures. 

"Strong  control  over  quality  is  an  asset  to  successful  performance  in 
production  planning  and  control.  Most  of  the  products  turned  out  in  a 
program  system  production  phase  are  intermediate  items  used  by  the  next 
phase  of  the  process.  Poor  quality  in  items  is  not  only  very  irksome 
to  the  personnel  who  must  deal  vith  them  but  create  errors,  inefficiencies 
and  work  delays  in  the  later  phases.  Having  to  rework  a  program  or  a 
document  can  disrupt  work  projections  and  other  work  may  be  stalled 
awaiting  the  results  of  the  re-do." 

This  quotation  Includes  all  of  this  section,  and  speaks  for  itself. 

It  is  of  sosm  Interest  to  consider  the  subject  paper  in  connection  with 

another  paper  by  the  sams  author  (Managing  the  Software  Development  Iffort, 

H- 21 395/000/00,  12  March  196U). 

This  chapter  is  directed  mainly  at  problems  of  costing  and  scheduling, 
but  contain*  acme*  material  relevant  to  the  quality  problem.  For  axampla, 
on  page  18,  we  find  a  graph,  "Instructions  per  Man  Month  as  a  Function 
of  Program  Length."  According  to  this  graph,  programmer  productivity 
flails  off  quite  sharply  when  the  total  number  of  instructions  in  a 
program  exceeds  about  200,000.  The  following  page  has  another  graph, 
"Computer  Hours  Used  as  a  Function  of  Program  Length,"  which  suggests 
that  the  number  of  computer  hours  required  rises  almost  exponentially 
as  the  total  number  of  instructions  increasts.  It  seems  reasonable 
to  guess  that  the  cause  of  this  Increase  in  computer  time,  and  lower 
productivity  of  programmers ,  arises  from  the  difficulties  of  maintaining 
quality  in  the  production  and  assembly  of  very  large  systems;  at  any 
rate,  the  quality  problem  can  certainly  not  be  divorced  tram  the 
production  problem. 

Study  Proposal  for  the  Development  of  Measurement  Variables  for  Air  Dsfenee 

ivaiuailon,  ITT  I.  Hillis  anA  mTs.  Sheldon,  W- 21735#  £  July  1$6». 

This  paper  is  the  last  of  a  series  of  four  dealing  with  proposed  measures 
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for  the  effectiveness  of  air  defense  crews .  It  Is  Included  because  such 
—lures  Bight  very  possibly  be  the  basis  of  quality  control  variables. 

In  fleet,  the  Measures  proposed  by  Hlllls  and  Sheldon  do  seem  to  hold  some 
previse  of  being  useful  for  quality  control,  though  the  exact  way  in 
which  they  would  be  used  is  not  discussed  by  the  authors. 

The  proposed  measures  include  such  things  as  detection  (yes  or  no),  detec* 
tion  latency  (time),  false  positives  (number),  detection  adequacy  (yes  or 
no),  maximum  tracking  error  (nautical  miles),  number  of  override  actions 
(number),  and  so  on. 

It  seems  perfectly  possible  that  the  list  proposed  by  Hillis  and  Sheldon 
might  be  used  for  reporting  quality  of  performance  of  a  system,  though  it 
Is  not  very  clear  how  they  could  be  applied  to  the  measurement  of  program 
quality,  nevertheless,  as  a  specific  proposal  naming  definite  variables, 
the  paper  seems  to  merit  consideration. 

Ifcs  Administration  of  Research,  Joseph  Pink,  SP-1684,  15  July  1964. 

Tbs  tltls  suggests  that  the  paper  might  discuss  evaluation  of  the  results 
of  research,  and  so  the  paper  was  examined.  No  discussion  of  evaluation 
is  in  feet  presented,  and  although  the  paper  is  interesting,  with  a 
number  of  cogent  commute  on  how  research  should  be  administered,  it 
does  not  seem  to  have  relevance  for  our  problem. 

Trig  heport  dated  31  July  1964  from  P.  B.  Tierney  to  J.  V.  Singleton. 

Tierney  says,  at  the  end  of  his  second  paragraph  and  the  beginning  of  his 
third,  "Is  there  a  manageable  number  of  variables  which  can  be  used  in 
evaluating  given  computer  facilities? . . .  .Economic  evaluations  of  computing 
systems,  large  and  small,  began  in  the  areas  of  strict  buslnsss  applies* 
tiocs."  Oh  page  4,  he  continues,  "There  is.  In  the  persons  of  both 
managsrs  and  consultants,  an  awareness  of  the  fluid  state  of  affairs  in 
the  measurement  of  performance  concerning  large  systems*  The  advent  of 
new  and  dramatically  changed  computers  will  affect  the  historically 
utilised  comparative  measures.  It  Is  the  purpose  of  these  individuals 
to  monitor  these  relative  and  absolute  measures  and  to  detenlne  whether 
or  not  they  are  deficient  or  superfluous  or  whether  re  rankings  and/or 
additions  are  required." 

Similar  comments  cover  the  first  five  pages  of  the  memorandum.  The  next 
24  pages  are  devoted  to  a  rather  detailed  description  of  the  computer 
equipment  available  at  8DC  Santa  Konloa;  this  includes  not  merely  a 
description  of  the  computers  themselves,  with  such  details  as  merry 
capacity,  cycle  time,  and  special  features,  but  also  s  listing  of 
peripheral  equip— t  available  with  each  computer .  There  is,  however, 
no  attmqt  la  these  pages  to  set  up  criteria  of  value  or  psrformsnc#. 
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The  Importance  of  the  document  llee  principally  In  the  fact  that  It  ' 
has  a  bibliography  at  the  end,  containing  98  titlea,  and  this  list  of 
references  would  presumably  be  of  considerable  use  to  anyone  undertaking 
a  further  study  of  evaluation;  however,  Tierney  at  no  tine  mentions 
program  quality  or  quality  control,  so  we  may  take  it  that  the  documents 
he  lists  are  not  highly  speicifc  to  our  problem,  but  at  best  shed  a 
marginal  light. 

The  Results  of  the  PD/MITHB/BPC  Work  on  Management  Control  of  Software, 
no  author,  MITRE  154-3551,  August  1964. 

This  document  is  included  because  of  its  title.  It  has  a  section  with 
the  heading,  "The  AFSC  Technical  Requirements  and  Standards  Progtam," 
but  does  not  contain  or  suggest  specific  quality  standards  for  software. 
In  fact,  it  reads  in  part,  "It  is  specifically  recceanudad  that  a 
follow-on  group  be  established  to  further  design  and  to  implement. 
Install,  train  for  and  maintain  staff  surveillance  of  the  process  of 
software  subsystem  acquisition."  It  is  not  very  clear  that  such  a 
group  would  be  concerned  with  product  quality,  though  on  the  following 
page,  the  writers  recognize  the  relevance  of  such  characteristics  as 
reliability  and  maintainability,  and  call  for  monitoring  of  tests, 
acceptance  standards  and  standards  application. 

The  document  does  not  appear  to  contain  any  material  useful  for  the 
present 'survey,  unless  we  so  regard  the  expressed  concern  of  1BD  over 
standards  and  testing. 

Cloaely  related  documents  with  overlapping  authorship  are  SDC  Experience  in 
Computer  Program  Implementation;  Coats  and  Cost  Factors,  L.  Fhrr,R(L)-1^93l/ 
602/OO,  30  June  1964,  and  Cost  Aspects  of  Computer  Programming  for  Command  and 
Control,  L.  Farr  and  B.  Manus,  8F-1372/000/01,  13  January  196^.  ""  ”~™ 

These  documents  all  have  about  the  same  orientation,  but  it  will  probably 
be  sufficient  for  anyone  interested  to  consult  7M-1W7/001/00. 

System  Installation  and  Testing,  Frank  V.  Hopkins,  KITTS  Publication  SR- 123, 
September  1964 

On  page  4,  Hopkins  says,  "Implementation  testing  should  be  deslgisd  and 
conducted  to  answer  the  following  questions: 

Does  the  system  as  installed  meet  the  spedflcatlonT 

Does  the  system,  meeting  the  specifications,  allow  the  job  to  be  done? 

How  well  does  the  system  do  this  job?” 

There  can  be  no  doubt  that  these  are  cogent  questions,  but  Hopkins  doss 
not  propose  any  operational  method  of  answering  them.  On  page  9,  for 
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example,  be  Mgr>,  "Injure  that  the  infomatlon  la  Interpreted  and  used 
properly  at  the  receiving  site.  Hoe  beat  example  here  la,  again,  that 
of  data  quality*  Presumably  at  the  receiving  site  there  are  toma 
conditions  under  vhich  we  wish  to  incorporate  cross- told  information  of 
good  quality  tracks,  other  conditions  under  vhich  ve  vlsh  to  incorporate 
cross- told  data  of  poor  quality  tracks,  and,  perhaps,  scan  conditions 
under  vhich  ve  vlsh  to  use  only  part  of  the  information  received."  Bare, 
the  definition  of  good  quality  and  poor  quality  seems  to  be  taken  pretty 
much  for  granted  and  our  raal  problem  is  to  make  these  definitions 
operational. 


lopklns  does  get  a  little  more  specific  later  on  in  his  paper.  On  page  18, 
under  the  heading  "Tests  to  tvmluate  Performance,"  he  aays,  "(Xu*  third 
general  objective  is  to  determine  hov  veil  the  system  does  the  job  for 
vhich  it  vas  bought. • . .Let  us  assume  as  a  desired  criteria  for  tracking 
continuity  that  the  track  should  be  vlthin  5  percent  of  aircraft  position 
90  percent  of  time  that  the  aircraft  Is  vlthin  radar  coverage  of  the 
environment,  vhlle  maintaining  the  earns  track  number  and  identification 
throu£i  out.  Our  measures  may  actually  indicate  that  for  95  percent  of 
the  time  the  track  heading  is  vlthin  2  percent,  the  track  speed  vlthin 
10  knots  and  tha  track  position  vlthin  1$  miles  of  the  corresponding 
pnrsmstsrs  of  tbs  aircraft.  This  information  is  useful  to  know.  It  can 
be  obtained  as  a  direct  by-product  of  testing  for  the  second  gsaeral 
Objective  and  can  be  collected  repeatedly  after  system  operations  begin." 
It  Is  dear  that  Bepklns  has  in  mind  the  reporting  of  such  quantities  as 
the  proportion  of  time  that  an  aatimatlon  error  exceeds  seme  assisted 
percentage,  sad  this  may  be  acceptable  for  system  operation,  but  the 
proposal  leave*  undetermined  the  question  vhether  these  particular 
deviations  are  in  fhet  tha  bast  ones  to  report,  or  hov  ve  should  arrive 
at  critical  percentages  that  would  be  taken  as  having  special  inferential 
value*  The  paper,  la  short,  doss  not  offer  s  specific  proposal  far  tha 
operation  of  quality  control  or  performance  evaluation* 

Wftfry&g:  3BT ay,t-  J.  1.  Cmkovleh  and  0.  Kail, 

this  deals  vith  parameter  testing  sad  points  out  scam  of  the  problems 
to  be  solved  in  connection  vith  parameter  testing  but  doss  not  deal  vith 
peowmm  production* 

In  oonaootion  vith  the  procoding  dooumsnt,  vs  have  M-21585  from  I.  I.  Wilmoarth, 
l 9  October  196k. 

This  Mmso  is  a  set  of  oemMnte  on  I(L)-21787  sad  rooords  seme  differences 
of  opinion  vith  the  authors  of  that  lots;  however,  it,  Jibs  the  Vote,  is 
concerned  vith  testing  a  program  after  It  is  produced,  and  net  vith  any 
evaluation  of  tbs  programming  prooesa. 
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Data  Processing  Task  Items,  Dellis  K.  Perry,  Memo  M-20963,  October  9>  1964.  * 

The  first  paragraph  of  thia  Memo  reads  as  follows,  "Attached  la  a  liat 
of  date-processing  task  statements  which  hare  been  prepared  for  use  in 
a  study  aimed  at  developing  better  understanding  of  the  data- processing 
Jobs  in  8 DC.  Individual  programmers  will  be  asked  to  describe  their 
Jobs  by  sorting  the  task  statements  according  to  the  Importance  of  the 
tasks  in  their  Jobs."  The  detailed  list  continues  on  the  following 
pages,  and  includes  a  number  of  steps  such  as  desk  checking  and 
parameter  testing,  but  is  merely  a  checklist  similar  to  the  one 
produced  by  Bolsky;  it  does  not  suggest  specific  criteria. 

I  think  the  document  is  not  helpftxl  for  discovering  quality  control 
procedures. 


Quality  Control  of  the  BUIC  Operational  Program:  Passive  Tracking. 

H.  I.  Stone,  HtzM  Mslioa-bion  W-070$l/01o4/d6/d/&>,  ton  fidential(  Title 
Unclassified),  16  October  1964. 

The  contents  of  this  paper  are  classified,  and  so  are  not  reported  here. 
However,  they  are  very  specific  to  the  BUIC  program,  and  hare  no  applloa- 
)  ticn  to  quality  control  in  a  general  sense. 

Controlling:  Time  and  Cost  Ihctore  in  Propamsini.  Robert  Bohrar,  OK  Report 
Volume  11,  10.  j,  a  publication  of  Computer  Sciences  Corporation, 

Deoember  1964. 

This  document  la  general  in  character;  its  main  purpose  eeeme  to  be  to 
serve  as  a  sort  of  salts  talk  to  prospective  clients •  Zt  describes  in 
qualitative  terms  some  of  the  things  that  (SC  personnel  do  in  approaching 
systems  problems,  and  even  has  a  facsimile  of  a  OK  form;  but  Z  don't 
the  paper  has  any  particular  value. 
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Data  Processing  Digest.  Volume  6,  No.  5,  March  1960/ 
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Control  of  Softvare,  MITRE  PI- 3551,  August  1$64. 
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BP-196,  November  19^2. 
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Corporation,  December  1964. 
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Dovst,  H.  See  Thomson,  V.  L. 

Doyle,  lauren  B.  Is  Relevance  an  Adequate  Criterion  in  Retrieval 
System  Ivaluationt  8^-126^,  I  July  1963* 
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